timeout
The page did not answer within the crawl timeout. Found by SEO crawler →5xx
The server returns an error instead of the page. Found by Technical SEO audit →
The page did not answer within the crawl timeout. Found by SEO crawler →5xx
The server returns an error instead of the page. Found by Technical SEO audit →
Requirements
SpiderHead runs on your own machine. The core needs Python and a few packages; everything else is optional.
- Python 3.10+
- macOS · Windows · Linux
- No hosted account
Components by operating system
| Component | macOS | Windows | Linux | Needs | Status | Notes |
|---|---|---|---|---|---|---|
| Core package | Supported | Supported | Supported | Python 3.10+, pip, git | Required | Installed into the project's virtual environment |
| CLI | Supported | Supported | Supported | Core package | Required | Ubuntu Server over SSH documented for headless use |
| MCP server | Supported | Supported | Supported | mcp extra, MCP SDK 1.29+ | Optional | Local stdio; any MCP client |
| Desktop app | .pkg | setup | tarball | Installer for each OS | Optional | Same core; scans shared with CLI and MCP |
| JS rendering | Supported | Supported | Supported | Chromium via Playwright, ~150 MB | Optional | Only for rendering checks |
| Screaming Frog CLI | Supported | Supported | Supported | Active SF licence | Optional | Only to drive live SF crawls; exports need nothing |
| Docker image | Supported | Supported | Supported | Docker | Optional | Slim 440 MB, full 1.08 GB |
Disk space
| What | Size | When you need it |
|---|---|---|
| Core with all extras | ≈ 860 MB | Virtual environment, measured on macOS with .[all] |
| Chromium | ≈ 150 MB | Only for JavaScript rendering |
| Docker image | 440 MB · 1.08 GB | Slim · full |
| Scan storage | grows with crawls | Keeps at least 1 GiB free; warns past 20 GiB of history |
Plan for about 2 GB
Core with all extras, Chromium and room for the first scans. Large sites need more: scan size depends on page count and whether bodies are kept.
Memory and budgets
Large crawls run under budgets you set: wall time, peak memory, database size. A budget hit stops the run as blocked, never as a pass.
Data sources
Search Console, GA4, Metrika, Webmaster, CrUX need your own credentials and are read-only. Paid providers are off by default.
No provider credentials are needed for the core crawl and audit.