Crawl Settings & Configuration

Site audits keep configuration deliberately small: what to crawl, how many pages, and how deep. Everything else - politeness, sitemap handling, noindex treatment - is automatic. Here's what each setting does and how to size it.

The Settings

URL. The starting point of the crawl. For a full audit use the homepage (or the section root you care about - the crawler follows links from wherever you start).

Audit type. Full Site Audit crawls outward from the start URL; Single Page Audit checks exactly one URL for 6 credits, with no crawl settings needed.

Pages to Scan. 1 to 500 pages, default 50. This is the hard cap on crawled pages and the main cost driver: 5 credits base plus 1 credit per page. Set it near your site's real page count - a 30-page site with a 500-page limit still only crawls (and bills) 30.

Crawl Depth. 1 to 10, default 3. Depth is how many link-hops from the start URL the crawler will follow: depth 1 is the start page and everything it links to, depth 2 adds what those pages link to, and so on. Most reasonably structured sites are fully reachable within 3; deep archives, paginated catalogs, or heavy category trees may need more.

Sizing the Crawl

• Small site (under 50 pages) - defaults are fine.
• Medium site - raise Pages to Scan to cover the real page count; keep depth 3 unless sections sit deep in the click path.
• Large site (500+ pages) - the 500-page cap becomes a sample. Point the start URL at your most important section, or run separate audits per section instead of one shallow site-wide pass.

The audit reaches pages two ways: by following links within your depth limit, and from your XML sitemap - sitemap URLs the crawl never saw are audited as well, and sitemap pages with no internal links pointing at them get flagged as orphans. A page that is neither linked nor in the sitemap won't be found.

What's Automatic

There are no controls for crawl speed, timeouts, or user agents - the crawler manages its own pacing against your server. Noindex pages are detected automatically and excluded from SEO checks, since they're intentionally out of search. URL variants that resolve to the same page (www vs non-www, http vs https, trailing slashes) are collapsed automatically so pages aren't audited twice.

Settings on Scheduled Audits

Schedules carry their own copies of Pages to Crawl and Crawl Depth, set in the Schedule Audits modal, so your recurring audit can be sized differently from ad-hoc runs. See Scheduling Regular Audits.

FAQ

Why did my audit stop at fewer pages than the limit?
The crawler ran out of discoverable links within your depth setting. Raise the depth, or check that the missing sections are internally linked or in the sitemap.

Does a higher depth cost more?
Only indirectly - cost is per crawled page. Depth just determines which pages get discovered inside your page cap.

Can I exclude a section from the crawl?
There's no exclusion setting; use the start URL to scope the crawl, and the skip/hide filters to keep known-irrelevant findings out of the results.

Related Documentation

• Site Audit Overview
• Starting Your First Audit
• Filtering & Sorting Results