GitHubDownload ZIP
301 chain301 chain
Permanent redirects stacked one after another.
Found by Redirect checker →
302302
A temporary redirect on a page that moved for good.
Found by Redirect checker →
Technical SEOUpdated

Search crawler

In short

A search crawler is automated software that requests web resources and discovers URLs before a search engine may render, index or rank them.

Mechanism and decision

A search crawler is automated software that discovers and requests URLs through links, sitemaps and prior knowledge. Fetching is only an early stage: a system may render, canonicalize, index or skip the URL later.

Important public URLs need direct internal links, predictable responses and a coherent index policy. Robots rules regulate compliant crawling, but a blocked URL can still be known from external links.

The diagram follows crawler discovery from internal links and sitemaps through fetch, render and index decisions.

What you can control

Crawling does not promise indexing, and indexing does not promise a result position.

Practical workflow

  1. State the public indexable set.
  2. Verify crawler identity before acting on bot-specific claims.
  3. Check robots, response, rendering and canonical signals together.
  4. Expose preferred routes through links and a current sitemap.

FAQ

No. Fetching, processing and including a page in search are separate steps.
Not by itself. User-agent strings can be imitated; use the search provider’s verification method for important log analysis.