Changelog 3.1.0

Roboto fonts and Material Symbols, and a single-column layout at 800 px.

document contracts, and display only core-backed scheduling data.

duplicate saves while the core is responding.

zone, competitors) and a Crawl tab (crawl mode, User-Agent preset or custom string, pause between requests, URL budget, SpiderHeadBot or a custom user agent); both read and write the project through the core, and the project overview shows the next and last scheduled run. A --tray mode runs the core's due scans and keeps the Mac awake only while a scan runs.

schedule (weekdays and/or month days, a time window in the site's time zone, enable/disable, competitors, crawl mode spider/sitemap/list, per-run URL and request budgets, politeness delay, and slice mode that resumes the previous partial crawl). It is edited through schedule-set / seo_schedule_set with expected_revision, and schedule-show reports the last run and next due window. schedule-run-due is a headless entry point that runs only due scans, refuses overlap with a lease, and records every decision in the project journal. schedule-install and schedule-uninstall write or remove a launchd plist, cron line or Task Scheduler command only when called explicitly.

googlebot-smartphone, bingbot, yandexbot, yandexmobilebot, baiduspider, duckduckbot, applebot, gptbot, claudebot, perplexitybot, and desktop and mobile browsers) or one custom string. The default is now SpiderHeadBot/<version> (+https://github.com/PavloSEO/spiderhead), and robots.txt group matching uses the same identity. Sites that matched the former SEOHEAD-Tools robots group must name SpiderHeadBot in their rules.

Each enabled check item resolves to one of three outcomes: succeeded when the saved audit ran the check, with the scan as evidence; unavailable when the crawl skipped or disabled it, with the crawl's reason kept; or unmeasured when the audit does not mention the check at all. An unmeasured item keeps its previous record and is reported by name in checklist_evidence, so it never counts as 0. A crawl that fails or is interrupted marks its automatic checks failed with the failure reason. The same recording path now serves the prepare flow, which previously carried its own copy of the loop. A project whose checklist is not initialized is skipped with a stated reason.

URL instead of the bare host, and a crawl of a competitor URL no longer marks the owner's checklist items; the result is recorded only in the competitor's own workspace. Opening a project with a damaged competitor workspace reports that competitor as unavailable instead of failing.

(project-competitors-add, project-competitors-list, project-competitors-remove) and the matching seo_project_competitors_* MCP tools. Each candidate keeps the URL as entered, including a www host, and www and bare hosts count as the same site for duplicate checks. A competitor with a www host no longer breaks preparation or makes project-observe fail.

and time budgets (3 requests and 2 seconds per page), so 50-300 page crawls are admitted without approve_large_crawl; budgets above the project thresholds are still refused.

stopped_by_budget and a finished_at time, instead of interrupted with no finish time. Its queued URLs remain for a resume.

stored without it.

per-site URL cap, and starts their scans in parallel; a competitor shows as scanned only after a finished retained scan.

evidence, so a missing or stale SF export no longer reports them missing; duplicate-check marks a run with zero compared pairs as unmeasured; site-wide template findings (same chrome heading, low text ratio) are one finding with a page count (#1314).

It no longer reconciles the whole built-in catalogue into coverage.json; built-in rows are added on demand when one is touched.

instead of calling the lock a leftover. The lock records its writer pid; a lock with no live holder is still reported as a leftover.

resumable, and a later pending step may be checkpointed past a non-success step. Steps still must be checkpointed in registered order, and an interrupted run still refuses further steps.

The item stays waiting for owner signoff or a non-agent reviewer.

concerns.

review; only an attempt awaiting a decision stays in review.

and a next_offset after the first 20 findings; findings_page pages through the rest.

own image fetches, and sitemap status (SITEMAP_URL_4XX_5XX, SITEMAP_URL_3XX, SITEMAP_URL_NON_INDEXABLE) from the crawl's own fetch of each sitemap-listed URL. A listed URL the crawl never requested is counted as unmeasured, never read as clean. IMG_MISSING_ALT, IMG_MISSING_DIMENSIONS and HREFLANG_ERROR say on a native scan that the crawl does not measure them as the SF export defines them, instead of pointing at an SF filter. Missing hreflang and mixed-content evidence on a native scan is reported as not measured by the native crawl, not as a missing export.

log.md, and the core records automatic entries for finished crawls (scans), built project reports and checklist attempts (agent). Entries go to events.jsonl and are read by project-event-page; the automatic hooks never fail the crawl, report or checklist step they describe. project-event-append and seo_project_event_append keep their interface and gain the log.md line.

(Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy). Before, they were dropped at capture, so a site that sent HSTS was reported as missing it. Set-Cookie headers store only their flags (secure, httponly, samesite=), never the cookie name or value. A security header sent more than once on the same page is reported as DUPLICATE_SECURITY_HEADER.

enough. The registered prompt defaults to skill:workflow/full-audit-v1 and the checklist task defaults to the one accepting the goal created (custom:goal-<entry>). Refusals name the owner command that fixes the missing link. The options are documented in seohead workflow-start --help.

the other keys default to unset. A refused population names its missing or unknown keys.

no sitemap. An absent fallback is recorded as incomplete but does not interrupt the crawl.

bounded per-IP records, visits, last visit and subnet filters. Explicit DNS verification checks each selected IP separately, records PTR and forward evidence, supports IPv6, and preserves unavailable/deadline outcomes as unchecked. Local caches expire after one hour; reads never trigger DNS.

from supplied provider prices, account for SERP depth and search-operator charges, and enforce known per-call USD ceilings before transport. Expand bounded backlink-index operations and journal missing costs or malformed response shapes as uncertain receipts. Project/month currency limits remain pending an owner decision.

full rel token observations, anchors, referring domains, filters and transparent spam review signals. Add CLI/MCP profile and reviewed disavow export routes without automatic submissions.

URL, with verified response/body lineage. Disabled retention, missing, invalid or over-budget bodies remain explicit omissions; partial findings carry coverage. Content-area exact hashes and bounded SimHash/LSH findings persist in the native audit without requiring an SF HTML directory. Threshold-adaptive LSH bands preserve qualifying-pair recall while reducing broad candidate buckets; near-analysis budget exhaustion does not discard exact-hash findings.

with an explicitly supplied local adapter or declared embeddings. No model or provider is selected automatically. Adapter identity, source hashes, scores, limits and omissions accompany review candidates; semantic similarity does not assert duplicate content, query relevance or cannibalization. The two semantic SF coverage rows stay gaps until a real adapter is selected and validated. External provider authorization requires literal boolean true; truthy strings or integers never authorize an embedding call.

source/anchor evidence, record redirect chains and loops with per-hop statuses, distinguish path-based internal search and empty anchors, and expose captured external no-response outcomes. Sitemap and robots consistency findings use retained membership and response evidence without live refetches; missing and partial evidence remains explicit.

adds the URL to the retained catalogue. Explained unresolved/excluded evidence stays immutable, while claimed known targets and non-null identity mismatches remain invalid.

was not stored as a crawled page, and marks unmeasured image responses as partial coverage.

changing the frozen scan SQL. Canonical links with unsupported hreflang/lang/media/type attributes, malformed annotations and cross-host/protocol declarations now produce evidence-bound findings; unknown well-formed relation keywords remain unclassified.

noindex/canonical observations. Native canonical chains and canonical/hreflang noindex findings can use probes; legacy status-only probes stay readable and unmeasured for HTML directives.

retaining numbered/stride rules and named unjudged states. Select saved finding URLs, including grouped members, through scan-url-query using temporary SQLite tables and streaming audit.v2 evidence.

truth is actually served and can be validated by native capture.

rows, preserving the frozen scan.v1 SQL schema. Structural AMP findings inspect the original doctype, head/body, UTF-8 charset, viewport, runtime script and boilerplate declarations; paired AMP targets without an AMP root marker are identified separately. Full AMP validator rules remain unavailable pending a licence decision.

heading jumps and the main landmark. Findings preserve bounded examples and full occurrence counts. Computed contrast and rendered axe-core rules remain unmeasured.

vocabulary and existing rich-result rules. Retained per-URL errors, warnings and missing required properties become native findings. Unsupported RDFa constructs are explicit partial evidence, not a clean validation result.

scan-structured-blocks; cap wide property trees and expose their limit. Read the latest document for a requested URL and representation without changing its scan.

Resolve supported JSON-LD identities against the same document base as HTML markup while preserving source JSON; reject scalar/empty JSON blocks as valid graphs and share normalized live JSON-LD tag selection across parser and served projections.

changing the frozen scan.v1 schema. Missing, incomplete and budget-limited bodies remain unavailable. Existing head resource detection and SF custom-extraction input are reused.

usage summaries and rendered image-sizing findings, and read them after reopening or offline reanalysis. Unsupported engines and missing measurements do not become clean results.

layout-shift culprits, image delivery, forced reflow and script execution findings, with capture provenance and input hashes. No Lighthouse process, paid provider or field-CWV inference is added. Refresh registry, SF coverage and generated reference documentation.

audit.v2 scan; previously it failed with 'NoneType' object is not subscriptable, because the audit lives in the companion rather than the scan's legacy table. The accepted record shapes are documented in seohead project checklist-record --help, the MCP seo_project_checklist_record schema, and docs/INPUTS.md.

retaining four verified test groups and combined coverage. Use the same stable Python version for CI and release-build bootstraps.

rows; Issues names every core check from the registry instead of showing "Other check".

time, so a checklist of any size no longer fails with "at most 100 rows". Desktop opens the scan a full audit leaves behind (largest complete scan) when the owner has not picked one, and core discovery also checks the desktop interpreter's own environment (sys.prefix) before PATH.

Preserve resource-budget settings without changing the frozen scan header, retain safe offline reanalysis and resume, and share native project-settings components, theme tokens and bundled fonts. Remove private design inputs from the distributed tree.

original scan header and source bytes.

and audit-workflow --action report attach the checklist snapshot to xlsx, docx, csv, md and PDF overview output instead of building an unbound report. A scan stopped by its URL budget (stopped_by_budget) is accepted as the project's latest audited scan and its report states the limited scope. report-build --input accepts the format key. report-build --project --format json on a streamed audit is refused with an explanation, because the JSON is written verbatim and cannot carry the checklist snapshot.

counters, a 10 px progress bar, the collector rate limit, and an honest 120 px speed-plot placeholder (core rate history is tracked in #1203).

against one saved HTML document, without a scan or a network request. Each result carries its values and a reason for an empty outcome (no_match, empty_html or budget_exceeded), so an empty answer is never presented as a clean result (#1059). It is a core call only; no CLI or MCP tool is registered yet, and fetching a live URL is deliberately not included.

one step covers most of a series' steps, a break from that stride is reported (?page=1,3,5,8 breaks at 5 to 8). Irregular offset schemes and unnumbered series remain unjudged and named as skipped.

separate mobile URL) into parse_html as mobile_alternates (#1051). The Mobile Alternate Link SF check is still a gap; the check that compares the mobile URL's canonical is not written yet.

against its rendered box and device pixel ratio (image_sizing: oversized and upscaled evidence). It is evidence only; no SF-analyzer check id yet, so "Incorrectly Sized Images" moves from gap to partial (#1049).

The semantic-similarity command and MCP tool exist, but no embedding adapter ships and the SF run does not call the module yet (#1041).

seohead/checks/search_console_findings.py joins crawl pages to a normalized GSC evidence document and evaluates GSC_ORPHAN, NO_SEARCH_DATA and NON_INDEXABLE_WITH_SEARCH_DATA. A check is skipped with its reason when the crawl is partial, the window is incomplete, the collection is sampled or truncated, or no evidence is supplied; a skipped check is never reported as clean.

analytics orphan URLs, indexable pages with no analytics data, non-indexable pages with visits, and bounce rate above 70%. Named columns --visits-column and --bounce-column select the data; a missing column is refused by name, unreadable values are counted and never read as zero, and analytics orphans are withheld on a partial crawl. Live per-URL Metrika/GA4 sourcing is not included yet (#990, #984).

Above 70%, No GA Data, Non-Indexable with GA Data) as found by crawl-enrich from an offline CSV instead of out of scope. sf run still does not emit them.

Playwright page.coverage payloads and reports per-resource unused characters, with a finding when a resource is mostly unused and large. Collecting coverage from a rendered page is the next slice.

AMP_MISSING_RETURN_LINK, AMP_NON_INDEXABLE_CANONICAL and AMP_INDEXABLE read the AMP target a desktop page declares with <link rel="amphtml">. Only AMP pages the crawl captured are judged; an uncaptured target is a named skip. The native crawl now records the resolved target as the AMP HTML column. The other ten SF AMP rules remain declined.

now raise NOODP and NOYDIR notices in the SF audit, previously listed as out of scope. none (noindex+nofollow) is covered by regression tests for the X-Robots-Tag path as well as the meta path.

response: MISSING_CSP, MISSING_X_CONTENT_TYPE_OPTIONS, MISSING_X_FRAME_OPTIONS (satisfied by a CSP frame-ancestors directive) and MISSING_REFERRER_POLICY. Pages whose stored headers cannot be parsed are counted as unmeasured, not clean, and a crawl without a stored scan skips them with a reason.

renamed archives) directly, detecting the format by content rather than file extension. A truncated or corrupt archive is reported as an error with no partial counts.

title_missing, title_over_60, h1_missing, noindex_meta) that AND-combine with the user's filters, and --export PATH with --export-format csv|xlsx writes every matching row to a new file. Export refuses above --export-max-rows (default 100,000) and never overwrites.

seohead/data/rich_results_rules.json, a versioned data file bundled with the package. The file's version is the value stamped as rules_version on every rich-result verdict, so changing a rule means changing data, not code.

now carry rules_version, so each result names the offline rule set that produced it.

(scripts/generate_project_catalogue.py, generate_registry_inventory.py, render_readme_visuals.py, linux_lifecycle_smoke.py, capacity_watchdog.py, profile_million_audit.py) in docs/README.md, with what each writes and which ones run in CI (#997).

receipts now live in seohead/projects/service_delivery.py (#995).

start page, as a column, filter and sort key. Pages no stored link reaches are NULL (orphans). crawl_depth is unchanged and remains the frontier depth, which is 0 for every sitemap-seeded URL. A scan that did not retain its links fails with reason_code: click_depth_unavailable instead of reporting a false NULL. The value is recomputed per query from the links table; persisting it as an indexed column at scan finish is still open because it needs a schema-version decision (#992).

ids, or all) that returns per-group counts (status_2xx–5xx, title_missing, canonical_missing, directives_noindex/nofollow, with_outlinks) over the same filters in one pass. Groups that need a projection (internal/external, sitemap, title duplicate/length, canonical self/other) are not included yet.

project-sources-unlink and project-sources-list (CLI, plus seo_project_sources_link|unlink|list over MCP) record which provider resource IDs a local project uses in sources.json beside project.json. Offline only: no discovery, no provider call, no credentials; project.json is never rewritten.

4xx/5xx, with the source page as evidence. It reads the stored resource graph and skips honestly when no image was measured.

after the crawl (5000 at most). A 4xx/5xx or redirecting target then reports CANONICAL_TARGET_ERROR, CANONICAL_TO_REDIRECT or HREFLANG_BROKEN_TARGET, as it already did for crawled targets (#987).

(details.path), so a fix can name each hop to replace. Refs #985.

details.inbound_links (internal links still pointing at the chain's first URL, with anchors, capped at 20) and inbound_links_count; without an all_inlinks inventory the field is null with a reason. Refs #985.

(semcore_project, semcore_cluster_count, semcore_landing_mapping). The semantic core stays in its own database and is not merged into the project database; detection never writes these facts, and a supplied fact cannot claim detection provenance. Documented in docs/PROJECTS.md under "Linking a semantic core" (partial #984).

The core does not yet return the whole link graph, so the canvas shows an honest waiting state and the filters stay inert until it does (#975).

(MCP) cancel a pending browser-consent flow record; status reports the pending flow when one exists. The flow record stores no grant material and expires after ten minutes. This is the first slice of #964: no browser start or callback yet, so no flow is created by the public surface and connected access is still never claimed.

run (mode=list), so it appears in project-observe and project-scans with its artifact in the project scans/ directory. A failed collector is recorded as failed; progress and resume of list runs are not covered yet (#962).

documents with a Python regular expression. Invalid expressions are rejected before scanning; a document that exceeds the regex time budget is reported unavailable, never absent.

journaled with cost_unknown and charge_status: unknown, so spend-report lists them as uncertain instead of counting them as a measured zero-cost call (#949).

request ceiling with other concurrent crawls of the same host through a private schedule in the user state directory (~/.config/seohead/.origin-pacing.sqlite). Projectless crawls previously ran unpaced across processes.

crawl-profile delete NAME removes one; both print JSON and refuse unknown names with the list of saved profiles.

crawler config as a named profile (refused when it carries credential headers); --profile NAME crawls with it, and --set still applies on top. Profiles live in ~/.config/seohead/crawl-profiles or the directory named by SEOHEAD_CRAWL_PROFILES. Listing and deleting profiles are not in this slice.

only_in_after, counts). When either crawl is partial, url_sets.unproven is true, so absence is not presented as proof.

response time, so a URL's chain reads from stored evidence without another request (#930).

that is not fetched again, so the repeated URL is visible in stored evidence; final_url still points at the last fetched hop (#930).

warning when the effective rate is above 2 req/s (the threshold is strictly greater than 2). The CLI prints the same warning to stderr; JSON on stdout and defaults are unchanged. The pacer was not touched.

bounded text, read newest-first in pages with source and text filters. Exposed as project-event-append / project-event-page in the CLI and seo_project_event_append / seo_project_event_page over MCP (issue #923).

Storage readers, the scan importer, the GSC archive, the sources store, the evidence-join store and the two checks classes now open their databases through one helper for connect, row factory and PRAGMAs. The two checks classes close their connections explicitly. Behaviour is unchanged; path normalisation and PRAGMA order stay at each call site.

Behaviour changes on purpose in these paths: - Redirect chains of exactly the hop cap (default 20) now resolve instead of being reported as unresolved. The loop check runs before the cap check, in both chain resolution and hop-path walks. - EEAT trust and citation checks compare destination hosts with their port (netloc), matching the internal-link host logic. Sites with explicit ports can change results. - llms.txt keyword checks match ASCII terms as whole words with an optional plural, so rapid, Showcase and Toolbar no longer pass. Cyrillic stems still match as substrings. - Navigation capture maps the reload reason to refresh_navigation. Unmapped requested reasons, such as pageBlockInterstitial, are recorded as frame observations that the validator accepts. - Technology detection treats a negated implication (["not", X]) as implying nothing. - Evidence import rejects duplicate declared metric names, and Search Console discovery uses the rows it is given. - Yandex Cloud retries sleep only between attempts. Yandex Webmaster query-analytics retries back off before the next attempt. - Schema build passes its timeout to the HTTP client (default unchanged). - The delivery receipt store is created with mode 0600 before SQLite opens it. - Semantic graph runs are serialised, because the stop-word set is module state. The semantic runner reports SQLite errors as ok: false instead of raising. - The CrUX error parser tolerates non-object response bodies. - Removed an unreachable sqlite3 is None guard in the ledger.

It shows the honest waiting state until the core stores a project link profile (#999); no sample rows, and the group filters are disabled with the same reason.

status text (shown only when a project-less crawl returns results), move the clear and save-as-project actions into an overflow menu below 1000 px, keep the address field and check-tab strip usable at narrow widths, scroll the results header instead of squeezing its columns, and give the shell status bar the same 8 px right padding in dark and light themes.

access are shown from real facts; core version and scan integrity show the neutral "unavailable in this core version" pill until the core reports them.

Window, and the finding and task context menus follow the canvas Menus board. Entries the core does not provide yet stay visible, disabled, with the reason in the tooltip.

and format choice, folder card, recent exports). Datasets and formats the core cannot export yet (compare, tasks, PDF) are shown as unavailable; the run button stays disabled until the export is wired (#1191).

draft review CSVs with the same apostrophe prefix already applied to =, +, -, and @. Both guards now share seohead.core.tabular.neutralize_formula.

Topvisor, BI destinations) no longer follow HTTP redirects, so an Authorization header is never re-sent to another host. A redirect now surfaces as a provider HTTP error.

default, SEOHEAD_RUN_LOG_MAX_BYTES overrides it): the previous generation is kept as runs.jsonl.1, and reads cover both files, so reuse lookups still see recent calls.

block tree, checks). The core returns only per-URL JSON-LD block counts, so the columns show honest waiting states tied to core issue 948 and the known counts stay real; no sample blocks are shown.

scan-status, scan-url-detail, scan-link-inspect and the project observer (project-scans, project-observe) no longer validate the whole artifact on every open: they check identity, schema, version and the header row, take counters from SQL (GROUP BY over pages_status), and say so in an additive validation field ("light" or "full"). Full validation runs on request (full_validation: true on scan-status, or --input '{"input_path": "...", "full_validation": true}' on the CLI) or when the same bytes (path, size, mtime, WAL state, header hash) were already fully validated in the process; a scan that fails the light checks is refused with the same error as before, and every writer or evidence-recording path still validates fully. scan-url-detail runs the redirect-target lookup (no index on responses.effective_url_id) under a 0.5 s budget and reports responses.redirect_target_lookup as skipped instead of scanning a large table.

incoming hyperlinks of a URL, looked up by address rather than document ID: anchor, rel with nofollow/sponsored/ugc flags, internal/external, stored position, target attribute and the HTTP status of the other end (null when that page was not scanned). Filters by link type, follow, status class and substring; offset/limit up to 200; total/filtered_total are exact or flagged capped. The scan is opened read-only without full validation, and failures carry a machine reason_code (url_not_found, scan_not_available, url_not_scanned, links_not_retained). Existing views and fields are unchanged.

seohead.mcp, seohead.tools is now seohead.checks, seohead.bot and seohead.remote_api moved under seohead.integrations, the CLI lives in the seohead.cli package (with seohead.cli.terminal_progress), the shared root modules (models, graph, filesystem, runlog, verification, canonical_policy, input_contracts, job_contracts, provider_matrix) moved to seohead.core, and build provenance moved to seohead._build.provenance. examples/ and config.example.json moved to docs/examples/; PROVENANCE.md and TRADEMARKS.md moved to docs/legal/. The seohead and sf-analyzer commands are unchanged; Python imports of the old module paths must be updated.

offline Screaming Frog exports, project plus agent), a feature map by area, a CLI overview, MCP setup for Claude Code and Claude Desktop, the companion desktop app and the measured-versus-not boundary, with real terminal screenshots under docs/assets/ captured from a local QA fixture and the synthetic export fixture. Add docs/CLI.md, which groups every registered command by job with its network/write/paid annotations (a drift test now fails when a command is missing), and docs/WORKFLOWS.md, which documents the previously reference-only workflow-run, inbox and monitor-record commands.

metrics, coverage and manifests; confirmed sampling, thresholding or truncation is partial. Search-versus-sessions cohorts require explicit collection quality, nonnegative count measurements and qualified source scopes, and retain each axis's source identity, attribution, search scope and calendar policy. The synthetic reporting package remains offline; native Sheets and Looker acceptance is still separate.

max_depth=-1 to disable the depth limit. Time/HTTP budgets remain separately configurable, storage guards and interruption/resume remain active, and legacy materialized routes refuse an unlimited URL population.

declared member when the configured limit is zero.

project-origin pacing lane as well as the per-scan limiter. Parallel scans still reserve one common host schedule instead of multiplying its rate.

retains the captured pages instead of leaking a KeyboardInterrupt traceback.

result. Preserve raw response data, source rows and exact returned counts in saved provider evidence. Link-count responses retain their documented page envelope and report multi-page populations as partial when only one requested page was collected.

and credential discovery. Keep account permissions and target access unverified, report malformed credentials explicitly, and preserve OAuth bearer support without requiring both sources.

routing boundary, distinguishing internal implementation from CLI/MCP availability without changing crawl behavior or capacity claims.

saved scan, optional follow-up phases, analysis and finalization. Unfinished captures retain their data and resumable checkpoint and return an explicit interrupted JSON result. An interrupt after the final commit preserves the completed artifact and reports its actual audit availability. Explicit resume completes unfinished post-collection work even with a drained frontier, preserving existing evidence-omission flags and retained sitemap inputs.

correction of older finished jobs, without changing retained artifacts or render budgets. Verify real queue backup/restore through authenticated status, downloads and persistent idempotency; keep the two-page recovery fixture on full rendering within its existing finite budgets.

handler has completed. Retained artifacts remain unchanged; a truncated output stream is never reported as successful.

project-local 2 requests/second dispatch gate for the same public host; expose each run's source kind, origin, config fingerprint, and aggregate rate boundary to observers.

longer injects the current spider configuration. Live spider and list runs continue to receive their explicit per-run profiles.

across operating systems, without changing production network policy.

renderer policy and URL provenance. Distinguish observed HTTP redirects, Chromium script/history/fragment navigation and unknown causes without asserting human clicks.

renderer settings; validate the earlier prerelease encoding without weakening provenance checks or treating observed events as renderer policy.

windows, and distinguish unmeasured counters from zero. Stream verified Screaming Frog progress while bounding both diagnostic pipes.

rehashing retained bytes on refresh. Explicit coverage reads keep byte verification; changed or legacy-unverified evidence remains visible as stale or unverified.

run, inbox and retained-evidence details with bounded pages and honest scope coverage. Keep storage reads and explicit note submissions off the input loop.

oversized paste without truncation, expose save receipts in every layout, and print unsaved drafts after a normal terminal exit for recovery.

materializing legacy JSON. Stream consistency findings, keep native page membership and finding deduplication on disk, and preserve exact anomaly counts while limiting only examples. Explicitly report the absent native exclusion-decision trace.

homepage was crawled. The exception requires matching sitemap summary evidence and does not exempt page findings, unrelated hosts, or unsupported target paths.

duplicating page and finding payloads in temporary SQLite storage. Full rows, deterministic plain/gzip exports, source identities and legacy inputs remain supported. Audit readers pin the validated snapshot through indexed row reads.

destinations; the manifest remains the final package commit marker.

full capability surface to its methods, inputs and evidence boundaries. Correct stale native/SF, project-coverage and remediation guidance; document the agent-authored developer workbook separately from factual report output, including reviewed acceptance criteria and source links.

Screaming Frog exports or an explicitly identified SF collector, and unknown collector. A native audit using the SF Analyzer schema no longer receives a Screaming Frog heading.

explanations in memory. Complete exclusion exports retain their order and source facts while keeping allocation bounded for large retained exclusion collections.

references against the public registries, and generate priorities from unresolved rows.

direct proxy manifests safely for targeted verification.

attribution on export, and retain spooled external-link checks across resume.

all extra and keep fake-browser tests independent of local Chrome selection.

dataset check, or staging write.

worksheet fixture passes the existing Sheets preflight and filtered-export consumers. Document the separate Linking API alias fields and require fresh-copy verification through the generated URL without manual source repair.

aliases used in URL parameters. Define shared run-field IDs, grouped provider date controls, and coverage charts that exclude overlapping populations. Native template creation and fresh-copy verification remain external acceptance requirements.

extraction, preserving repeated occurrences separately and labelling partial populations.

URLs through a durable SQLite evidence artifact, and stream that artifact into BI without materializing page or matched-URL lists. This does not deliver a copyable Looker Studio template.

a bounded disposable SQLite index. Keep dynamic provider dimensions valid through selected exports and offline Sheets/BigQuery plans; bound partition counts and disk consumption.

inside the retained source JSON, so exact URL selections conserve affected findings.

distinguishing a known missing return from an uncrawled or body-unavailable target. Report observed noindex alternates, preserve raw labels and normalized URL identities with duplicate/conflict context, and mark partial population relationships unmeasured. The existing audit, CLI, MCP and report surfaces carry the same findings without additional fetching.

thresholds while retaining URL/origin, form-factor and collection-period scope. Missing, incomplete and failed field evidence stays unavailable or partial.

constructed remote API and worker service.

2,000,000 HTTP request budget and the existing storage-only synthetic profile. CLI, MCP, remote queue and workers preserve the same declared budgets. Materialized legacy/list APIs retain an explicit 50,000-URL safety bound; list robots, retries, redirect and canonical hops now share their request budget. Native audits and offline reanalysis use the bounded audit.v2 path without the former 10,000-page or 20,000-form bridge refusal. Transactional writer counters avoid rescanning the growing frontier on each batch. A manual acceptance driver records loaded source identities, interruption/resume and downstream consumers; admission is not itself evidence of completed million-URL acceptance.

interrupted artifacts, verify every CSV/XLSX row and task coverage, and exercise guarded owned-loopback HTTP with unchanged default pacing separately.

default and independent project, request, disk, lease and render budgets. This allows a million-page target to fit its unchanged polite request rate; values above the duration ceiling and invalid types fail before collection.

preserving query interruption and cancellation without charging Python parsing or derived writes against the original scan-opening validation deadline.

sidecars during collection and resume instead of retaining duplicate in-memory lists. Checkpoints verify sidecar counts; older v4 inline-form checkpoints migrate without losing form findings. The existing audit bridge names its page/form/link materialization bounds and preserves collected evidence when an audit is unavailable. A prior report is retained under a stale filename rather than presented as the current run. Native JavaScript escalation reads page records through a re-iterable SQLite view instead of rebuilding all page objects. Full large-audit streaming remains tracked by #816/#817.

availability.

without rebuilding every route or loading every node into Python. Long routes retain their exact depth and explicit truncated endpoints.

Derive nonempty corpus and hreflang relations through disk-backed indexes, retain complete evidence collections in audit.v2, and label bounded relation finding samples with their exact occurrence counts.

oversized pages remain readable and mutations invalidate their old cache weight. Keep HTML size ranks as stable scalar URL/size pairs instead of retaining every full page model, and avoid retaining serialized page JSON in mutation callbacks.

trailing-slash pairs, and close durable adapter and remote-job SQLite connections explicitly.

state instead of candidate page populations, and process stored-HTML DOM matches in two passes over the fixed file index. Preserve normalized-variant representative ordering, existing URL examples, complete matched-source coverage and every finding.

corrupt or unterminated evidence without replacing it. Compare older inline form checkpoints with their sidecar without allocating a second form list.

order, multiple hosts and no-discovery semantics. Stream exchange-file input, preserve request/time budgets across collection resume, retain bounded response/hop evidence and keep graph/form checks explicitly unavailable. Large directory lists use the same spool with a separately bounded JSON audit bridge.

populations, and keep snapshot and prune operations paired with the companion while preserving the scan.v1 schema.

revision 5, with backward-compatible audit reads, atomic ledger migration, explicit member paging through CLI/MCP, and complete source-bound BI membership companions.

group scope and close ingest resources on failed validation.

response, document, body, or resource evidence, avoiding repeated whole-corpus reads on every sparse page commit without changing scan storage.

remote context and browser on cancellation or disconnect without local fallback.

references, with redacted retained provenance and failure before collection when a reference is unavailable.

while keeping its endpoint outside submissions and preserving pinned, policy-bound browser requests.

record conservation before a rendered report is published.

evidence and review gates. Interrupted runs can be resumed by a later local session only after their current dependencies remain valid.

triage receipts for task, goal, competitor, blocked, or rejected outcomes.

against their SHA-256 receipts. Missing or changed evidence makes prior completion stale without rewriting history; verification budgets never silently certify evidence or discard its receipt.

Passive workflow observation checks retained verification metadata without rehashing artifacts; changed or unverified receipts stay explicit.

ratios (#794). project-checklist-init / seo_project_checklist_init accept an optional plan — a reviewer, a URL population declared as a complete_set, a named sample, or unknown, with its source, size or enumerated urls, and optional per-templates populations, plus an optional tasks agreement (all_agreed, or a sourced selection of existing checklist item IDs) — bound to the project site identity and stamped with the write revision. Recording a plan upgrades the checklist to seohead.coverage.v3 and appends to a retained plan history; re-recording an identical agreement is an idempotent no-op, a changed agreement stale-marks earlier evidence instead of re-evaluating it under a changed denominator, and single-plan v3 documents migrate on read. Reading, reconciling, or applying priorities never upgrades or invents a population, and v1/v2 documents keep loading unchanged. Items outside a selection are reported as not_agreed, visible outside every denominator.

by a limited (partial-scan) measurement stays unfinished instead of counting as complete, while an item scoped to explicit sample URLs still finishes when the artifact covers exactly that scope. The top-level complete flag now means the scoped audit is finished — every applicable agreed task done, no pending exclusions, and the agreed URL population fully covered by verified measurements — so unknown or unmeasured scope can never report a completed audit.

(with measured_urls), manual_review, deliverable_review, and url_population — each with its own basis, numerator, denominator and state, so a URL count is never mixed with a check or task count. A denominator that cannot be justified stays null with state: "unknown" and a reason instead of becoming 0/0 or 100%; measurements that cannot be verified against the agreed population keep the URL axis partial and count only verified eligible URLs. URL membership is verified against the agreed enumeration, so a population declared only by size counts no measured URLs and the numerator can never exceed its denominator. Per-template populations are validated as agreed sub-populations of the site population when a plan is recorded or read: enumerated template URLs must belong to the enumerated site set, and declared template membership can never exceed the agreed site size, so an incoherent plan is refused rather than trimmed or counted past the denominator. Saved check evidence now carries the measured URL enumeration, so the URL numerator is an auditable set rather than a row count. Multi-site aggregation sums the axes per site and degrades to unknown whenever a site cannot justify its denominator.

and an inspectable evidence basis — a project-relative digested artifact or an explicit evidence reference. Reviewed exclusions keep their stable ID and history, leave the task denominators only through that recorded decision, and are listed separately under exclusions. A disabled item, a stale exclusion record, or a legacy not_applicable record written before the evidence-basis rule stays inside the denominator as pending_exclusion until reviewed again, so disabling, catalogue removal, or reclassification can never silently shrink agreed counts; a fresh reviewed exclusion on a catalogue-removed item resolves normally instead of staying pending forever.

evidence hashes, including streamed audit.v2 baselines.

per-task denominators and bounded case pages for large ledgers.

results-affecting policy; a reused scan UUID alone can no longer resolve an older case.

separate future-service health, proxy, TLS and firewall boundary.

crawl settings, project-scoped permissions, idempotent submission, OpenAPI schema and scan-status-compatible evidence. It requires separate durable workers and target/egress policy; local CLI/MCP installs remain unchanged (#784).

evidence-normalize reads a supplied CSV/XLSX/JSON export or a saved seohead.provider-evidence.v1 envelope fully offline and produces a versioned seohead.normalized-evidence.v1 document against an optional seohead.evidence-mapping.v1 manifest, keeping per-field provenance (envelope, declared, or unknown) for source, reporting identity, period, timezone, attribution, engine, collection coverage, and URL binding. A measured 0 stays zero while missing rows, null or blank cells, formula cells, suppressed values, uncollected periods, and failed or skipped collection stay explicitly unavailable. evidence-join joins normalized evidence to a scan, audit, or page list under the strict external_join.v1 key policy with explicit relaxations only, preserves matched, crawl-only, external-only and unkeyable populations at their declared grain, and reports key collisions instead of multiplying rows; --compare plus a declared policy adds a pure compatible/incompatible/unknown decision across period, timezone boundary, identity, attribution, engine, and grain, and reports measured-zero quadrant eligibility for downstream cohort work. Restricted sources keep rows and identifiers inside a private local artifact and return counts only.

navigation, evidence detail, saved-view/review state, and explicitly scoped MCP inbox notices.

preventing an automatic second upload when acceptance cannot be established.

population counts, coverage, and profile; report missing renderers and oversize inputs before sending. Match the thorough wizard JS budget to the shared queue contract.

deduplicate trusted updates, and isolate status, cancellation, recovery, and delivery by initiating actor/chat. Identical intentional reruns remain separate jobs.

begin response by its deterministic stage-sheet identities before retrying.

worksheet targets and dataset contract drift, bind recovery to the exact configured target, and retain resumable staging after quota exhaustion. Budget target-grid growth before the atomic replacement and report unconfirmed row counts on interrupted runs.

with selected findings, and disclose local-only membership references in destination and workbook receipts without claiming the companion was uploaded.

row-conservation Partitions sheet instead of failing at Excel's per-sheet row limit.

.xlsx.index.json with exact record ranges, source population, selection and coverage references. Preserve the previous report on caught write failures and clean streaming spools on interruption.

them, protect Summary metadata from formula injection, and reject malformed retained audit rows before an incomplete export can claim success. Complete CSV/JSON and the bounded PDF overview policy remain available.

including skipped-check reasons, and support exact finding selection by retained crawl scope segment.

and optional service delivery restart-safe.

requests and by actual response Content-Type before parsing or body retention. Both filter stages record their decisions while preserving link, status, header and redirect evidence; filtered HTML metadata remains explicitly unmeasured. Script/stylesheet resources keep their independent selection and byte budgets.

coverage instead of treating parser defaults as clean or as observed missing metadata.

discovery.external.crawl runs a separate phase after the internal frontier closes, fetching each distinct outlink target once under its own external_checks.max_targets/max_hosts/max_requests/max_depth/max_redirects budgets — never recursive off-host traversal. Outcomes (fetched, failed, blocked, skipped with the rule that declined) append to external_checks.jsonl in the run directory, keep source-page provenance, persist across resume without reopening spent budgets, and report their own coverage in discovery.external_checks distinct from internal crawl completeness. Requires discovery.external.store; refused by name on the native SQLite route (--scan-out) until that collector gains the phase.

of hard-coding Chromium: rendering.browser.engine selects chromium (still the default), firefox or webkit through the same guarded Playwright path, an unknown name fails during settings validation rather than at launch, and a requested engine whose binary is not installed reports a dependency error that names the fix (playwright install <engine>) instead of silently falling back to Chromium. Firefox's documented limitation is enforced honestly: it rejects rendering.browser.mobile_emulation (is_mobile is unsupported there) while the independent rendering.browser.touch_emulation (has_touch) still reaches the context. Probes and full renders share one effective configuration, so a pattern is never escalated by a browser that differs from the one producing its evidence.

viewport dimensions alongside the existing desktop/mobile presets, which keep deciding the profile exactly as before. The pair is all-or-nothing and bounded to 1..16384 px; a custom pair replaces the preset's dimensions only -- it does not imply mobile or touch emulation, which remain explicit flags an operator states.

during escalation, separately from speed.concurrency, which keeps governing HTTP crawl politeness. The default of 1 preserves the sequential behaviour and builds no pool at all; above 1, probes and renders run on a bounded worker pool while every verdict, commit and SQLite write stays on the orchestrator thread in submission order. The render deadline, URL budget and sampled/full policies hold under concurrency, a stopped batch waits out the pages it already started and never schedules new ones, and interrupted or failed workers leave no thread, page, context or browser alive behind the call.

configured page concurrency are recorded in the crawl manifest and fingerprint and in every rendered document's renderer provenance -- including failed attempts, which record what was requested without inventing a successful render. Both escalation paths honour the settings: the directory crawl and the native SQLite scan, where rendered-link candidates still enter the frontier through the existing deduplicating admission once rendering.rendered_links.crawl is enabled.

availability ranges, and persist quota waits and bounded transient retries.

complete fresh export set before analysis. Full-profile provenance records the verified export manifest and crawl-rate readback; an explicit rate limit fails before crawling when its base config is missing or invalid.

"Query is too complicated" refusal is now retried in calendar-month slices with a 5–15 s backoff; a slice that still refuses is resubmitted at a sampled accuracy=0.1. Only metrics provably additive over disjoint periods (ym:s:visits, ym:s:pageviews) are merged — a query for unique-visitor, percentage, average, or unverifiable metrics is refused before the first slice request rather than returning a summed value that would be wrong. Slices are merged as they stream in under one global row cap that also bounds downloads: once the cap is exhausted, remaining partitions are never requested and their date spans are named in split.unfetched — a slice stopped mid-way lists its own span there with partial — whole-period total_rows/totals stay unknown when the unqueried periods make them unknowable, and the returned window is marked partial. Merged answers re-apply the caller's sort (dimensions and metrics, or the API default of first metric descending) and take the offset/limit window from the global merged set. Sampling fields reflect what the API actually reported (sampled, worst sample_share), and a slice that does not report them leaves the whole-period value unknown — null in the CLI/MCP answer, never an invented false. contains_sensitive_data is the OR over slices, min/max are recomputed on merged rows, and missing or malformed metric cells and totals mark the body incomplete instead of reading as zero. The split block keeps the requested and resolved periods, the timezone used, and per-slice accuracy, attempts, dropped rows, and sampling metadata; a partition that fails names the completed spans and the accuracies exhausted. Relative dates (today, NdaysAgo) resolve in the request timezone or the counter's own zone (time_zone_name, else its minute offset) — never the host date. A period that cannot be mapped to dates (a shorthand such as month) is still retried and sampled before the error is re-raised. An explicit accuracy is honored in every documented form (low/medium/high/full or a numeric share such as "1"): anything above the last-resort sample keeps the accuracy=0.1 fallback, while a caller's already-lower setting (low or a share at/below it) is preserved rather than resampled upward. Every attempted request — including the refused whole-range call — reaches the spend journal.

/indexing/samples instead of the nonexistent /search-urls, new broken_links_samples reads /links/internal/broken/samples, and paging honors each endpoint's documented page-size cap (500 for queries, 100 for the sample lists) instead of sending limit=500 everywhere (#713).

shape — including a count below the returned rows — requests at most the remaining row budget per page so a max_rows cut can never overread into a false complete, exposes returned/truncated for a single non-paginated page as a bounded slice, reports partial whenever rows remain unread, and fails malformed pages instead of claiming a complete empty result (#713).

subrequests, and reports certificate verification failures without transport detail.

fixture-CA trust, POST replay, response caps and private-target rejection through an owned synthetic CONNECT interceptor without changing platform trust.

frequency request contract (#720, #779).

base is broad (WS), quoted is the quoted phrase ("WS"), overal is the exact wordform (!WS) !W asks for, and exact is the strict-order form ([!WS]).

quotes made it reject the whole batch with 422 JSON_VALIDATION_ERROR. Rewritten phrases are reported back in cleaned so a caller can see which ones were measured in a different form.

payload. A phrase whose result names only other regions is omitted and listed in warnings instead of borrowing another region's numbers.

ready and the new components field now come from the same credential components (OAuth bearer, durable grant, or service account) instead of only the bearer file, so a configured service account no longer reads as "GSC is not set up". The service-account component now validates the JSON document itself (private regular file, bounded size, service_account type, the exact Google token URI, and required fields) and both doctor blocks report a safe service_account_status enum (missing, unsafe_file, too_large, unreadable, malformed_json, unsupported_shape, or configured_unverified), so an oversized, unreadable, or malformed file stays distinguishable from a configured-but-unverified one without exposing its contents.

accepted frontier row reaches a terminal state, including runs whose retained evidence is partial: crawl_partial already names omitted observations, so a drained queue no longer reads as an interrupted capture in scan-status, scan-list, or history (#712). An explicitly stopped run -- aborted before it accepted work, cancelled, or halted by the error circuit -- keeps interrupted and its recorded reason even after the queue drains, including through resume_or_finalize, so stop paths never receive a false finished timestamp.

instead of dying on normal-size sites: the scan's open-and-validate deadline is budgeted by artifact size, and each document read gets its own deadline, so a 1.5k-page corpus no longer ends in interrupted. The corpus input-byte budget meters what the analyzer actually retains — extracted Markdown for duplicates, one digest per page for boilerplate — not raw HTML size, and reaching a corpus bound reports partial coverage with the exact prepared/eligible counts and the bound that stopped the read (#710).

instead of forwarding it to the Reporting API, which answers offset=0 with HTTP 400; new offline tests pin the 1-based contract across metrika-report, seo_metrika_report, and provider-collect aggregate_report, and verify that paginated stop conditions (total_rows, ROW_CAP) count the rows skipped before a non-default offset (#707).

now started without waiting and stopped once the PDF is verified.

summary, diagnostics, sqi_history, search_history, query_history, in_search_history, events_history, indexing_history, important_urls, broken_links_history, external_links_history; params for dates and indicators; paging of query and crawl-sample lists (#713).

request sent with offset=0 with HTTP 400 (#707, #728, #729).

a browser consent script that stores a read-only Search Console grant.

report all-failed sitemap expansion as ok: false.

without retaining cookie values; credentialed and persistent-profile crawls remain non-resumable.

--input and --json-input continue for one release with a warning.

scans/ directory by default. --out-dir remains the explicit legacy directory route, while list input and cache live/replay configurations name that route rather than silently changing collection mode.

when no source checkout is available.

packaged audit skill with permanent/temporary intent and canonical-target guidance, citing Google Search Central. Temporary redirects remain valid for temporary changes; a status code alone does not measure a ranking loss.

its raw and browser requests. An explicit --user-agent is available through CLI and MCP, and every completed or partial comparison records the effective identity and viewport size.

titles and saved reproduction evidence instead of registry identifiers or collector names. Historic records without sufficient evidence state that limitation explicitly while JSON audit artifacts remain unchanged.

second pass rather than a parity standard.

or newer.

embeds and marketing counters, and retained links to generated references for the live interface surface.

accurate retained-corpus and browser/provider boundaries.

offline-reanalysis contracts (#666). The documented late-page field table is now checked against the importer mapping, so an older JSONL observation remains SQL NULL instead of becoming a fabricated clean value.

checks as skipped when their source fields were not collected.

rendered_bodies in a crawl-site result reports how many rendered DOMs were retained and the reason for each one that was not, and the CLI prints that beside the finish line rather than leaving it to a capability flag that says "partial" for one missing DOM and for every one of them. A run that discarded nothing prints nothing extra. The counts are read back out of the scan artifact, so a finished scan still answers the same question afterwards.

rules.check_og lists og:url among a page's missing_tags whenever the field reads falsy, and the native path never read the tag at all: no og_url on PageRecord, no og_url column in scan_v1.sql, no OG:URL in the Internal:All projection crawl/evidence.py builds. The column existed only on the Screaming Frog side, where normalize has always resolved OG:URL, so an export of a site saw the real value and a native crawl of the same site reported a defect -- an unmeasured field presented as a measured one, on every page of every native crawl, whatever the markup said. The tag is now read where the other three are (the parser already returned it as og:url), stored in a nullable pages.og_url -- nullable like every other late-added column, so a record written before it was collected stays distinguishable from a page that declares none -- and projected as OG:URL. This is the collection side of the seam #646 fixed the read side of. tests/test_open_graph_seam.py gains a third fixture site whose pages have Open Graph but no og:title, so the finding actually fires and its list can be read: one page declares og:url and must not be named missing it, the other declares none and must be. The test that matters audits both of those pages twice, once from the crawl and once from an Internal:All export describing the same two pages, and requires the same verdict -- #646 and #654 were both two paths disagreeing with no test spanning them.

html_bodies in a crawl-site result reports how many fetched HTML page bodies were retained and the reason for each one that was not, and the CLI prints that beside the finish line rather than leaving it to a capability flag that says "partial" for one missing body and for four fifths of them. A run that discarded nothing prints nothing extra. Redacted request and response header lists now keep the names of the headers redaction removed, under x-seohead-redacted-headers and never their values: two responses whose bodies were treated differently used to record byte-identical header lists, so an operator reading the artifact could not tell why a body was missing.

changelog.d/, so a pull request no longer conflicts with every other pull request that landed before it (#638). Every branch added its entry at the top of ## Unreleased, which made that one line the file's contention point: nine hand-resolutions across eight pull requests in two sessions, each a re-push and a fresh CI run, and the resolution was "keep our block, then theirs" every single time. A conflict whose answer never varies carries no information and is a chance to drop a paragraph while thinking about something else. The .gitattributes fix the issue proposed -- CHANGELOG.md merge=union -- was measured and rejected before this: union merges line by line, so when both entries contain an identical line (a blank line inside an entry is enough) it keeps one copy and reports success, deleting the other entry's line with no conflict and nobody prompted to look. A first test with deliberately distinct lines merged cleanly and hid that. Trading a loud conflict for silent data loss is strictly worse than the conflict, so the structural fix is the one that shipped.

unique to a branch by construction: one pull request closing two issues writes two fragments, one per issue, and a change that closes nothing writes no-issue-<slug>.md. A fragment holds the entry exactly as it belongs in the changelog -- one or more top-level bullets with their continuation lines, no front matter and no wrapper -- so the multi-paragraph prose entries this file carries round-trip byte for byte rather than being reformatted into one-liners. python scripts/build_changelog.py folds them into ## Unreleased at release time, ordered by issue number rather than by whatever order the filesystem hands back, and --prune removes the fragments it folded in. Assembly replaces the whole region between two marker comments instead of inserting into it, so running it twice produces the same file both times whether or not the fragments were pruned in between. CHANGELOG.md itself is not regenerated on every pull request, deliberately: a gate that kept it continuously in sync would put both branches back in the same file and rebuild the conflict this removes.

checking anything is worse than the conflicts. Both were found and both now cover the fragments: tests/test_docs_drift.py carried CHANGELOG.md in PUBLIC_MARKDOWN, so the English-only gate, the "references only existing commands" gate and the stale-count gate now read changelog.d/*.md as well, and the two count-drift gates skip a fragment for the same reason they skip the changelog -- an entry records what was true when it was written. tests/test_doc_counts.py deliberately excludes the changelog from its live-count scan and its file list never reaches changelog.d/, which is the same answer for the same reason; tests/test_doc_commands.py, tests/test_docs_commands_execute.py and tests/test_examples_gate.py never read the changelog at all. A new tests/test_changelog_fragments.py pins each of those answers, along with fragment naming, deterministic order, idempotent assembly, and the fact that two simulated concurrent branches adding a fragment each merge without a conflict.

The pinned render route read the origin with httpx's undecoded stream and handed those bytes to Playwright together with the origin's content-encoding. route.fulfill never applies a content coding to the body it is given, so Chromium parsed gzip as text/html and built a DOM out of the compressed stream: no title, no h1, no canonical, no links, and a rendered size equal to the transfer. Every site that compresses its HTML -- nearly all of them -- came back ok: false, reason: incomplete_render, and the message blamed the render, suggesting --wait domcontentloaded or a longer --timeout when the render had finished perfectly. seohead.tech reported 22 435 rendered bytes against 121 793 raw; it now reports 123 048 with the real title and 31 links. The route now reads the decoded entity, drops content-encoding from what it forwards (content-length was already dropped as hop-by-hop, so Playwright states the length of the body it actually received), and asks the origin only for the codings this client can decode rather than forwarding Chromium's own invitation to br and zstd. A coding that arrives undecoded anyway aborts the request and names itself, because passing it through would reproduce the bug in silence. The byte cap now counts decoded bytes, which is the body the renderer sees -- a compression bomb previously passed the cap compressed.

parse_html stores each tag under its full property name -- {"og:title": ...}, as its own docstring says -- and crawl/collect.py asked the same dict for "title", "description" and "image". dict.get answered None every time, so og_title, og_description and og_image were empty on every page of every native crawl ever run: 0 of 40 920 pages carried one on the run that surfaced this, on a site whose pages do publish Open Graph. That is the repository's central rule inverted -- an unmeasured field read as a measured one -- and it travelled, because crawl/evidence.py projects those columns into OG:Title, OG:Description and OG:Image of the Internal:All frame, so a check saw blank on a native crawl and the real value on a Screaming Frog export of the same site. check_og's honesty contract hid the damage rather than showing it: with no OG column populated anywhere it skipped, so a site with Open Graph audited identically to a site without it. The parser is unchanged; only the read side moved. A new tests/test_open_graph_seam.py crawls two loopback fixture sites and follows the value from parser to pages.jsonl to the pages table of the scan artifact to the OG:* frame columns, crawls a second site with no Open Graph to hold "empty because absent" apart from "empty because unread", and pins the parser's key shape in the same file -- the failure was two correct modules disagreeing about a key name with no test spanning both.

157 to 161 checks and adding an internal-linking skill (#634). The pieces existed -- every edge with its position, anchor and nofollow flag in the links table, ORPHAN_PAGE, INLINK_BOILERPLATE_ONLY, LOW_LINK_SCORE -- and nothing turned them into an answer about the graph. A 40 920-page audit carried the wrong headline for hours because of it: 66% of the sitemap was reported unreachable by internal links, a figure produced by subtracting a crawl's URL count from the sitemap's, which measures "the crawler did not get there". Walking the artifact's own 5 252 235 edges from the home page put reachability at 91.9%. The real defect was depth: 23 742 pages -- 58% of the site -- were more than ten clicks from the home page, and the deepest was 3 005, because the archive is a linked list of "next post" links rather than a tree. Click depth is now the headline the audit prints. summary.internal_linking carries the histogram, the maximum, the reachable/unreachable split, internal edges by page position, and how many edges repeat another edge, and audit.md renders it beside the health summary. The walk starts at the URL the crawl actually began from, never at pages.crawl_depth: that column records where the crawler happened to reach a URL, and a sitemap-seeded crawl records 0 for most of a site, so a walk seeded from it describes somewhere else. When no start URL was recorded and more than one page claims depth 0, the measurement refuses to run and says which.

DEEP_CLICK_DEPTH fires on an indexable page whose shortest route from the start URL exceeds thresholds.click_depth_max (default 10, and every finding names the number it used). DUPLICATE_INTERNAL_LINK fires on a page that writes the same destination and anchor more than once: on the site above, 509 744 of 1 134 307 edges -- 44.9% -- were repeats, and they were one masthead emitted twice with the second copy removed by JavaScript, a template defect that had taken a hand-written script to find. LINK_INSIDE_HEADING and IMAGE_LINK_WITHOUT_TEXT need what a link's page region cannot carry -- the anchor's own ancestor chain and contents -- so a native crawl now records a small, capped per-page link_placement_json beside the heading outline, and both checks skip by name on a Screaming Frog export rather than run clean on nothing.

An edge with no recorded position is counted as unclassified and never folded into content: link_position.classify defaults to off, which leaves that on every edge, and a run in that state must read as "nobody looked" rather than as a site whose links are all body copy. A position absent from the distribution is reported as "no edges classified here", not as a region the site lacks -- rules are tried in order and a menu inside <header> matches nav first. A partial crawl withdraws the click-depth verdict whether or not it fired, because a frontier never fetched may hold the shorter route, and a check silently finding nothing would read as "every page is within the floor".

155 to 157 checks (#632). The eight existing heading checks judge headings as a set: they can say a page has one H1 and four H2s, and nothing more. So a page whose DOM order is H2, H2, ..., H1, H2 satisfied every one of them, and so did a page whose H2s were all menu labels in its masthead -- found on a live site as an article template with eighteen headings standing before its own H1, reported as clean on headings. What was missing was the evidence, not the rule: pages stored h1, h1_2 and h2 -- three strings with neither order nor place. A native crawl now records the whole h1-h6 outline in DOM order, each heading with its level, text and page region, in a new nullable heading_outline_json column, and HEADING_BEFORE_H1 and HEADING_IN_PAGE_CHROME read it. The region reuses the taxonomy link classification already uses (nav, header, sidebar, footer, content, other), so "in the header" means the same thing about a heading as it already does about a link. A heading no position rule matched is placed against the content area only when the document named one -- a <main>, a [role=main], an <article> or a configured selector; where the content root falls back to the whole <body>, calling a heading "content" would mean nothing more than "somewhere on the page", so the region is recorded as unmeasured and the page is named among the run's skipped checks with that reason rather than passed as clean or flagged as a defect. Order needs no region, so HEADING_BEFORE_H1 still answers for those pages. A Screaming Frog export carries no outline at all, and both checks skip there by name.

EMPTY_ROBOTS -- the stand-in _fetch_robots returns on every path where robots.txt could not be read -- was still the parsed-robots shape from before the parser grew groups and sitemaps. The success path returns parse_robots's shape, and that is the only shape the scan artifact's robots_summary context accepts, so in --scan-out mode a crawl aborted with native parsed robots summary is invalid whenever robots.txt was unreachable, answered 429 or 5xx, redirected off-host, into a loop, past the hop budget or to a non-text/plain body -- or was a plain 404, which is not an error at all but the RFC 9309 "no restrictions" case and the most common of the seven. Both documented outcomes were lost behind that one line: a missing robots.txt now crawls unrestricted again, and an unavailable one stops the run with the note saying which of the six ways it failed. The legacy directory-mode crawl never noticed because nothing reads the old keys. Found on a live site whose robots.txt 302s to a cookie-sync endpoint for any user agent it does not recognise.

A native crawl printed its effective request rate and then nothing at all until it finished; on a 34 000-page site the only way to tell it was still working was to watch the scan file grow in ls -la. The line now refreshes in place on a TTY with pages fetched, the URLs known so far, the percentage of that set, the current request rate over a trailing window, elapsed time, and the scan artifact's size when --scan-out is writing one. What it deliberately does not do is imply a total it cannot know: a crawler discovers its own workload, so the denominator is fetched + queued capped at the URL budget, it is labelled known rather than total, and a header line printed once says the frontier is still growing and that the percentage is not an estimate of when the run will finish. An unmeasurable rate reads as rate n/a, never as 0.0 req/s. Both numbers come from the crawl's own structures -- for a SQLite scan, from the same resume_snapshot query the collection loop steers by -- so what is on screen is what the artifact holds at that moment rather than a parallel tally that can drift from it. A non-TTY (a pipe, a log file, CI) gets a plain line every 30 seconds instead of carriage-return redraws, and a new -q/--quiet silences both the progress line and the startup rate line while leaving the JSON result on stdout untouched.

checks. PAGINATION_MULTIPLE and PAGINATION_URL_NOT_IN_ANCHOR report a page that declares two different successors, or one whose declared URL is not also an anchor on the same page. Successors are compared after URL normalization, the same identity the anchor half uses, so /blog/page/2 and /blog/page/2/ from two plugins are one successor spelled twice rather than an ambiguous series. PAGINATION_URL_NOT_IN_ANCHOR needs a page's whole link inventory and reads the All Inlinks export for it; PAGINATION_MULTIPLE does not, and now reads the lighter Internal:All rel="next" 2 / rel="prev" 2 occurrence columns first, the same way CANONICAL_MULTIPLE already answers from Canonical Link Element 2 — All Inlinks is only a fallback for a profile whose Internal:All was written without those columns. PAGINATION_SEQUENCE_ERROR reports a break in a page-number run the series otherwise follows, per the issue's own caveat: a series may start at a number other than one, a stride is not a break, and a series whose URLs do not state a page number is declared unevaluated rather than judged against a numbering that would have had to be invented. Every series left unjudged is named among the run's skipped checks with the reason true of *that* series -- a stride, a cycle, a series too short to hold a run and an unreadable page number are four different statements, and three of them describe series whose URLs all state their number. The count is per series, not per run, so the ordinary WordPress shape (page one at an unnumbered /blog/, the rest at /blog/page/N/) cannot disappear behind one judgeable series elsewhere on the same crawl.

geo_guard checked only the advisory country string, so search_volume/keyword_ideas/keyword_difficulty/serp could still reach DataForSEO's live endpoint for location_code=2643 (Russia) or 2112 (Belarus) whenever a caller supplied the numeric geo-target without also filling in country; the guard now checks location_code first, since that is the field actually billed on. Separately, a network-level exception (URLError/TimeoutError/SSLError) during a billed call — DataForSEO's post, Arsenkin's /set, and Yandex Cloud's wordstat.topRequests/wordstat.dynamics/ web.searchAsync — used to retry the identical payload with no idempotency key and log only the attempt that finally returned a response. None of the three providers offers an idempotency mechanism for these endpoints, so a lost response is no longer retried: the attempt is recorded in the spend ledger (cost unknown, flagged attempt_failed: network_error) and the call fails outright instead of risking a second charge. Idempotent reads (Arsenkin /check//get, Yandex Cloud operation polling) are unaffected and keep retrying. yandex_cloud.WebSearch.search_batch isolates a lost submission to its own query instead of aborting the batch. metrika.py retries network errors the same way but is not billed money and was left unchanged; flagged for a follow-up if its Logs API export creation should get the same treatment for quota reasons.

TOCTOU window described in #14, protected only collect.py's list-mode fetch — the other fourteen call sites, including spider.py's crawl-site engine, let httpx resolve DNS a second, independent time to open the socket, so a hostile resolver answering that second lookup differently than the guard's own reached 169.254.169.254 or any RFC 1918 address regardless of the guard's verdict. http_client() now builds every client on a transport that pins the connection to the address it resolves itself, on the first request and on every redirect hop, keeping the hostname only for the Host header and TLS SNI — so the fix is structural rather than a discipline every caller had to remember. SEOHEAD_ALLOW_PRIVATE_NETWORKS/SEOHEAD_ALLOW_PRIVATE_HOSTS are unchanged.

arrives with (#120). Which scenarios exist is decided by the coverage map rather than by taste: each declares the catalogued issues it resolves, and tests/test_scenario_coverage.py asserts that every issue this toolkit claims to find appears in at least one, that no scenario names an issue the map does not list, and that none omits its limits. All 143 findable issues are covered. Every command in every file is still executed against the fixture site in CI — the suite runs 2008 tests, of which the doc-command gate is now the largest single group.

entries in one file, SITEMAP_TOO_LARGE above 50 MB uncompressed, and SITEMAP_URL_DUPLICATED for a URL declared in more than one sitemap, naming both documents. A file over either limit is invalid rather than merely large — a search engine may read part of it and discard the rest with nothing the site owner can see — and the thresholds come from the protocol, not from config. Both parsers now record the per-document byte size and declared count, and the findings name the child sitemap rather than the index. Registry grows from 118 to 121 checks.

(#115). sitemap.crawl() returned only the normalised loc, so a sitemap declaring /a/ caused a fetch of /a — on most CMSes a 301 the crawler invented and then reported as a fact about the site, and a 404 wherever the slashless form is not routed at all. Each entry now carries loc as published alongside loc_normalized for the consumers that compare on it. Redirect statistics from earlier sitemap-seeded crawls are suspect for this reason.

.claude/skills/ now has two tiers: 21 method skills, each covering one thing well, and control/, which decides which of them to run on a site nobody has looked at yet. The controller routes rather than restating, and carries five loadable sub-skills (scoping, rate and load, reading an audit, verifying, deliverables) and a three-file reference archive (defects found on live sites and what gave each away, which population each check describes, and what the toolkit cannot answer at all). The English-only gate and the doc-command gate now cover every Markdown file under a skill directory, not only SKILL.md.

Comparison still happens on the normalised key, but a finding that named a normalised form named a URL appearing nowhere in the crawl — unactionable for a reader, and indistinguishable to the anomaly scanner from a finding about a page never fetched.

_DispatchGate now reads the crawl's injected clock instead of time.monotonic() directly, so the test drives it with a virtual clock that advances only when something sleeps: the dispatch instants are the crawler's own arithmetic and the assertion is exact. The old form compared real elapsed gaps against a 0.024s floor and failed on unchanged code whenever the machine was busy.

http.headers is merged into every request beside the credential headers; speed.adaptive gates the throttle's delay and concurrency adjustment (the timeout and server-error counters keep running — giving up is a separate mechanism from backing off); discovery.hyperlinks.store / .crawl, discovery.external.store and discovery.redirects.crawl now decide what the crawl records and what it follows. discovery.canonicals.*, discovery.external.crawl and discovery.redirects.store are removed rather than wired: they named capability the spider does not have (canonical-chasing, cross-host crawling) or state it cannot withhold, and a setting that appears in --config-help and the run manifest while changing nothing is worse than no setting. The coverage canary's exemption set is now empty: every DEFAULTS path changes an observable outcome and is named by a test.

(#96): main, then [role="main"], then article, recording which one matched as auto_main / auto_role_main / auto_article. The previous default — the whole body minus the nav and footer tags — counted 126 template words out of 433 on a live WordPress post (29%), including a skip-to-content link, and that inflation feeds THIN_CONTENT and LOW_TEXT_RATIO in the same direction on every page of a template. header and aside join nav and footer in DEFAULT_EXCLUDE_TAGS for the fallback path. A configured selector still wins, and one that matches nothing still falls back to fallback_default_body rather than silently auto-detecting a different region.

It compared a sitemap's URLs against every destination in the crawl's link graph, so on a live 124-page site it fired 392 times — 362 image files a gallery links to directly, five off-host links, and 30 URLs the crawl never fetched — which was 74% of that report and buried the findings that were real. The observed side is now the pages a sitemap is supposed to declare: fetched, 2xx, HTML, same-host and indexable. reconcile_sitemap takes that population as a separate comparable argument, so SITEMAP_ORPHAN keeps asking about reachability against every link destination and cannot invent orphans; what was set aside is returned under linked_not_comparable rather than dropped.