Crawl Budget

Meaning and scope

Crawl budget is a practical label for how much crawling a search engine is willing and able to spend on a site. Google describes crawl demand and crawl capacity as important parts of the process. It matters most on large or frequently changing sites, where duplicate URLs, errors and weak internal discovery can compete with pages that actually need recrawling.

zoom_in

How to inspect it

Begin with URL populations, not a universal crawl quota. Compare the canonical indexable inventory with the URLs found in logs or Search Console reporting where available. Look for parameter combinations, redirects, soft errors, thin paginated paths and orphan pages. A low crawl count by itself is not evidence of a problem if the important set is already found and refreshed.

1. Define the preferred indexable URL set and keep it separate from useful but non-indexable utility URLs.
2. Crawl representative templates to find duplicate patterns, redirect chains, errors and non-canonical links.
3. Make important pages discoverable through standard internal links and a current sitemap.
4. Reduce accidental URL generation before asking for more crawling capacity.
5. Recheck logs or search reporting after changes and label unavailable evidence as unavailable.

Practical decisions

Prioritise URL quality and serving reliability. Consolidate duplicate routes, fix broken internal paths and make canonical pages directly reachable. A sitemap is a declaration of preferred pages; it is not a request that guarantees crawling. Changes should be staged and measured because broad blocking rules can hide pages that need discovery.

Limits and mistakes

Crawl budget is not a dial for small sites and is not a substitute for helpful content or indexability. Do not block a URL merely because it looks unimportant without confirming whether it supports rendering, navigation or canonical discovery. A faster server may improve crawl capacity, but it does not make every generated URL worth crawling.