Automated Page Discovery
Stay aligned with what is live
Stop pasting URLs by hand. We discover candidate pages from sitemaps and structured crawling so your monitoring set tracks what is actually published, including routes beyond the homepage. Tune scope, add manual URLs, and refresh as sites change when launches and content churn outpace static checklists.
Why it matters
Less manual upkeep, fewer blind spots, you stay in control.
Sitemap-first
Parse sitemap.xml, follow indexes, and respect priority and changefreq where present.
Crawl when needed
HTML crawl fallback for same-domain links when no sitemap exists (depth, caps, and patterns configurable).
You approve the set
Discovered pages sync as candidates; add, remove, or reprioritise before they drive quota.
How discovery runs
-
We resolve sitemaps, collect URLs, mark auto-discovered candidates, and sync them to your site. Crawls stay domain-restricted and respect robots.txt.
-
Ideal when marketing ships pages faster than ops can update spreadsheets.
Scope controls
| Control | What it does |
|---|---|
| Include / exclude patterns | Limits which paths enter the monitor set |
| Max URLs / depth | Caps crawl size and recursion |
| Manual overrides | Add or pin URLs outside discovery |
From discovery to scheduled tests
Discovered pages feed the same priority and schedule model as manual URLs, so coverage grows without someone enqueueing every new route by hand. Scheduled testing & priorities explains cadence and queue behaviour.