What is indexing
Indexing is the process that follows crawling. If crawling is simply reading a page, indexing involves parsing the content, extracting key signals (headings, links, structured data), and storing them in the index. Only indexed pages can rank.
How to control indexing
- Sitemap.xml — tells the bot which pages are important for indexing.
- Meta robots (noindex) — prevents indexing of a specific page (useful for carts, filters, admin sections).
- Canonical — points to the main version among duplicates, consolidating indexing signals.
- X‑Robots-Tag — controls indexing via HTTP headers (e.g., for PDFs or images).
- GSC → URL Inspection — manual request for indexing or re‑indexing.
Indexing control tools
- Google Search Console: Coverage report, URL Inspection tool, submit to index.
- Yandex.Webmaster: similar features for Yandex.
- Screaming Frog SEO Spider (with GSC API integration) — bulk indexing status checks.
Indexing is not instantaneous. Even after submitting via GSC, it can take from a few hours to several days. For new sites, the first pages may take weeks to be indexed.
FAQ
What is the difference between crawling and indexing?
Crawling is the process of a robot visiting pages to read content. Indexing is analyzing and adding the page to the index. A page can be crawled but not indexed (e.g., due to noindex or low quality).
How can I speed up indexing of new pages?
Use GSC for manual URL submission, add pages to sitemap.xml, create internal links from already indexed pages, and build external links.
Does noindex affect crawling?
The noindex tag does not block crawling. The bot may still visit the page, see noindex, and drop it from the index. To completely block access, use robots.txt (but then noindex won‘t be read).