blocked by robots
robots.txt keeps crawlers away from the page. Found by Crawl scope and robots →no sitemap
No sitemap.xml found at the usual places. Found by Sitemap crawler →

Guide

From install to the first report, then the app, agents and data sources. Eight short pages; full reference docs stay on GitHub.

Let your AI agent install itRecommended

Install SpiderHead for me.
1. Clone https://github.com/PavloSEO/spiderhead into ~/spiderhead.
2. Create a Python 3.10+ virtual environment inside it and run python -m pip install -e ".[all]".
3. Install Chromium: python -m playwright install chromium.
4. Check it works: seohead --version.
5. Register ~/spiderhead/.venv/bin/seohead mcp as an MCP server in this agent's configuration.
Tell me what you did and anything that failed.

Paste it into Claude Code, Codex, Cursor or any LLM agent that can run shell commands, in any folder.

Download ZIPThe latest version of the repository as a ZIP. Unpack and follow “Install SpiderHead”.spiderhead-main.zip

Or with gitgit clone https://github.com/PavloSEO/spiderhead.git

Installers for macOS and Windows are coming. Until then the ZIP always holds the latest code.

Start here

about 15 minutes

1Getting startedWhat SpiderHead is and the shortest path to a report.3 min 2Install SpiderHeadmacOS, Windows, Linux; pip or uv.5 min 3Your first crawlCrawl, read the issues, export.7 min

Go further

Analyze Screaming Frog exportsAn audit and a task list from existing exports, offline.Read Reports and task listsXLSX, DOCX, PDF and a developer backlog.Read The desktop appIssues, URLs, link graph, projects and tools.Read Connect AI agents over MCPClaude Desktop, Claude Code, Cursor, Codex.Read Set up data sourcesSearch Console, GA4, Metrika, Webmaster, CrUX.Read Full reference on GitHubTOOLS.md, CHECKS.md and every command.Open docs

Stuck? Every tool whose extra is missing answers with the exact install command instead of crashing. Still stuck — open an issue on GitHub.Open an issue