7 min

Your first crawl

crawl-site follows links from a start URL on the same host, respects robots.txt and audits the result with the same checks used for Screaming Frog exports.

Start small
The URL cap is an explicit budget, not a claim about site size.
bash
```
seohead crawl-site --url https://example.com --max-urls 500 --scan-out ./scans/audit.sqlite
```
Configure
Every setting can be set in a JSON file passed with --config or with --set; --config-help lists them.
bash
```
seohead crawl-site --config-help
```
Read the result honestly
Each check reports a finding, ran without findings, skipped or failed. Skipped checks name the missing evidence; partial crawls withhold link-graph conclusions.
Resume or diagnose
Crawls are resumable. crawl-diagnose explains what stopped a crawl.
bash
```
seohead crawl-diagnose --scan ./scans/audit.sqlite
```

Edit this page on GitHubUpdated 10.10.2026

Start small

Configure

Read the result honestly

Resume or diagnose

Check your redirects locally