Screaming Frog SEO Spider is desktop crawling software that requests site URLs and reports links, responses, metadata, directives and crawl paths.
A crawl is a bounded snapshot
A crawler shows what its configuration can request at a point in time. Start URLs, JavaScript rendering, limits, user agent, cookies and exclusions determine the dataset.
It can identify a redirect chain or missing title, but it cannot prove indexation, conversions or an algorithmic cause.
Configure the crawl
Write scope before pressing Start: canonical host, crawl limit, rendering mode, rate and excluded routes. Retain the configuration with the export.
- Define scope.
- Choose rendering.
- Cap rate.
- Run to completion.
- Record failed or skipped URLs.
Inspect findings
Open representative findings in a browser and inspect headers before prescribing a fix. Compare crawl URLs with sitemap or Search Console only after normalising redirects and URL variants.
| Finding | Verify live | Do not infer |
|---|---|---|
| 3xx chain | Final response and links | Ranking loss amount |
| Canonical conflict | Rendered tag and target | Indexation state |
| Orphan candidate | Crawl versus inventory | That every unmatched URL is defective |
Common errors
A raw export is not an implementation plan. JavaScript mode, authentication and URL normalisation can all change the answer.