Crawler Settings

The QuantSearch crawler discovers and indexes your website content. Configure it to match your site structure and performance requirements.

Basic Settings

Max Pages

The maximum number of pages to crawl per job. This is limited by your plan:

Max Depth

How many links deep to follow from your start URL:

• 0 - Only crawl the start URL(s)
• 1 - Start URL + pages directly linked from it
• 3 - Recommended for most documentation sites
• 5+ - For deeply nested content

Concurrency

Number of parallel workers (1-20). Higher values crawl faster but put more load on your server. Default is 5.

Content Filtering

Include Patterns

Only crawl URLs matching these regex patterns. One pattern per line.

# Only crawl /docs/ section
^/docs/

# Only crawl English pages
^/en/

Exclude Patterns

Skip URLs matching these patterns:

# Skip preview/draft pages
\?.*preview=true

# Skip private areas
^/admin/
^/internal/

# Skip generated files
\.pdf$
\.zip$

JavaScript Rendering

Enable JavaScript rendering for Single Page Applications (SPAs) or sites with dynamically loaded content. This uses a headless browser to render pages.

Custom Headers

Send custom HTTP headers with each request. Useful for:

• Basic authentication (staging sites)
• API keys
• Custom user agents

{
  "Authorization": "Basic YWRtaW46cGFzc3dvcmQ=",
  "X-Custom-Header": "value"
}

Single URL Crawls

You can also crawl specific URLs without following links. This is useful for:

• Refreshing specific pages
• Adding new content immediately
• Testing changes

Enter a set of URLs (one per line) in the "Crawl URLs" modal on the Overview tab. This is a one-shot crawl — it runs once and does not change the site's saved settings. For a large or persistent list, use a Saved Crawl Scope below.

Sitemaps

Point the crawler at one or more sitemaps in the site's Settingstab, under Sitemap URLs (one per line). Relative paths like/sitemap.xml resolve against your site. Sitemap index files are followed automatically.

When you save a sitemap, QuantSearch checks that it is reachable and looks like XML. If it returns an error or is not a sitemap, you get a warning. The crawl still runs, but it falls back to crawling from your site root instead of the sitemap's URLs — so confirm the path is correct.

Saved Crawl Scope

A Saved Crawl Scope is a persistent list of the only URLs a site should crawl. Set it in the site's Settings tab. Unlike a one-shot Single URL Crawl, it is saved and applies to every future crawl, including scheduled ones. There is no limit on how many URLs it holds. Leave it empty to crawl the whole site.

Robots.txt

By default, the crawler respects your robots.txt file. It identifies as:

User-agent: QuantBot/1.0 (+https://quantcdn.io/bot)

To allow QuantSearch while blocking other bots:

# robots.txt
User-agent: *
Disallow: /

User-agent: QuantBot
Allow: /

Scheduled Crawls

Set a recurring crawl in the site's Settings tab, underCrawl Schedule. Choose Off, Daily, Weekly or Monthly. The first run happens one interval after you save it, not immediately, and the schedule shows its last and next run times.

Available frequencies depend on your plan:

• Free - scheduled crawls are not available
• Pro - Weekly or Monthly
• Enterprise - Daily, Weekly or Monthly

Every crawl consumes page updates from your monthly allowance, so the frequency you choose and the size of your index together decide how much of it you use. A 50,000-page index crawled daily is 1.5 million page updates a month; the same index crawled monthly is 50,000. Indexing pauses when the allowance is used, so pick a cadence that matches how often your content actually changes.

A scheduled crawl uses the site's saved configuration, including any Saved Crawl Scope or Sitemaps above. Suggested cadence:

• Documentation - Weekly, or re-crawl after each release
• Blog - Weekly
• Marketing site - Monthly, or re-crawl when content changes

Troubleshooting

Pages not being indexed

Check:

• URL matches your include patterns (if any)
• URL doesn't match exclude patterns
• Max depth allows reaching the page
• Page returns 200 status code
• robots.txt allows crawling

Content not extracted correctly

If content appears wrong in search results:

• Enable JavaScript rendering for SPAs
• Check if content is behind login/authentication
• Ensure content is in standard HTML (not canvas/images)

Crawl too slow

• Increase concurrency (be careful with your server load)
• Disable JavaScript rendering if not needed
• Use include patterns to focus on important sections