Starting a New Crawl
Learn how to configure and start a website crawl, including all available options for different subscription tiers.
Starting a New Crawl
Crawling is the first step in automated testing with AegisRunner. The crawler discovers pages on your website, detects forms, and gathers information needed to generate comprehensive test suites.
Plan Limits
| Plan | Crawls/Month | Pages/Crawl |
|---|---|---|
| Free | 15 | 25 pages |
| Starter ($9/mo) | 150 | 100 pages |
| Pro ($29/mo) | 500 | 500 pages |
| Business ($59/mo) | 2,000 | Unlimited |
Accessing the Crawl Page
There are two ways to start a crawl:
- Click Crawl in the left sidebar
- Click the Crawl Website quick action on your Dashboard
Crawl Modes
AegisRunner offers two crawl modes:
Crawl Settings
Basic Settings
| Setting | Range | Description |
|---|---|---|
| Max Pages | 1-1000 | Maximum number of pages to crawl (limited by your plan) |
| Depth | 1-10 | How many links deep to follow from the starting URL |
| Include Patterns | URL patterns | Only crawl URLs matching these patterns (e.g., /blog/*) |
| Exclude Patterns | URL patterns | Skip URLs matching these patterns (e.g., /admin/*) |
Crawl Options by Plan
Different features are available based on your subscription plan:
Free Tier Features
- Fill Forms - Automatically fill detected forms with test data
- Submit Forms - Submit forms to discover more pages
- Skip Auth Forms - Avoid submitting login/registration forms
- Accessibility - Run accessibility checks on discovered pages
Pro Features Pro+
- Record HAR - Capture HTTP Archive files for network analysis
- ARIA Snapshots - Capture accessibility tree snapshots
- Error Snapshots - Screenshot pages with JavaScript errors
- Responsive Test - Test pages at multiple viewport sizes
- Dark Mode - Test dark mode appearance
- SEO Audit - Analyze SEO factors on each page
Business Features Business
- Reduced Motion - Test with prefers-reduced-motion enabled
- Forced Colors - Test high contrast mode compatibility
- Memory Leaks - Detect potential memory leak issues
- WebSockets - Monitor WebSocket connections
- Security Audit - Check for common security issues
- Offline/PWA - Test Progressive Web App functionality
- Geolocation - Test with simulated geographic locations
- Timezone - Test with different timezone settings
- Network Throttle - Simulate slow network conditions
Starting the Crawl
Make sure you have the correct project selected. The crawl will use the project's base URL.
Select Full Site or Single Page based on your needs.
Adjust max pages, depth, and enable desired features.
Click the Start Crawl button to begin.
Real-Time Progress
Once started, you'll see real-time updates via WebSocket connection:
- Current page being crawled
- Number of pages discovered
- Forms detected
- Errors encountered
- Progress percentage
Crawl History
Below the crawl configuration, you'll find your Crawl History showing all previous crawls with:
- Status (Completed, Running, Failed)
- Start URL
- Pages discovered
- Duration
- Timestamp
Click on any crawl to view detailed results.
Best Practices
- Start with a small max pages limit to test your settings
- Use exclude patterns to skip admin areas, logout links, and delete actions
- Enable "Skip Auth Forms" to avoid accidentally submitting login forms
- For large sites, use include patterns to focus on specific sections
- Run crawls during off-peak hours to minimize impact on your server
Common Exclude Patterns
Consider excluding these URL patterns:
/logout*
/signout*
/admin/*
/delete/*
/api/*
*.pdf
*.zipRelated Documentation
- Understanding Crawl Results - Analyze your crawl data
- AI Test Generation - Generate tests from crawl data
- Test Data Management - Configure form fill data