What is Crawl Budget
The limit of resources that a search engine allocates to crawl your site. Especially important for sites with 10k+ pages.
Why it Matters
If the crawler wastes time on duplicate filters or session URLs, it may not index new useful products. Optimizing budget means blocking trash in robots.txt and improving internal linking.
Factors Affecting Budget
- Site popularity — more backlinks → higher budget
- Server response speed — slow responses reduce budget
- Amount of low‑value pages — duplicates, filters, sessions
- Content freshness — frequent updates increase budget
How to Optimize Crawl Budget
- Block useless pages in robots.txt (but not from indexing, from crawling)
- Use noindex on trash pages (filters, sorts, sessions)
- Ensure fast server response (caching, CDN)
- Internal link important pages so the crawler reaches them quickly
- Properly configure XML Sitemap — include only canonical, useful URLs
- Avoid infinite spaces (calendars, pagination date loops)
Blocking pages in robots.txt does not save budget if there are external or internal links pointing to them — the crawler will still attempt to fetch them, see the block, and waste time. For budget efficiency, use noindex or URL parameter blocking in GSC.
FAQ
Can crawl budget be an issue for small sites (under 1000 pages)?
Usually not. Google can easily crawl 1000 pages in one go. Indexing problems on small sites are caused by other factors: server errors, robots.txt block, or poor internal linking.
How can I find out my site's crawl budget?
Google does not disclose an absolute number. You can infer it from Google Search Console → Crawl Stats report. Look at the trend — how many requests Googlebot makes per day and how many pages it crawls.
What if my crawl budget is too low?
Improve server speed (reduce TTFB), block useless URL parameters in GSC settings, remove duplicates, set up clean internal linking, and refresh your XML Sitemap.
Does crawl budget affect pagination?
Yes. If you have many pagination pages with non‑canonical parameters (page=2, page=3…), Google may crawl them. Use rel='prev'/'next' (deprecated but sometimes works) or block deep pagination via URL parameters.
Why does Google crawl my old pages but not index new ones?
Likely budget goes to outdated or low‑value content. Check for duplicates, endless filters, session URLs. Block unnecessary dynamic parameters using the URL Parameters tool in GSC.