Crawling
In one line
Crawling is the act of search engine bots following links to fetch and read pages — the step that has to happen before anything can be indexed or ranked.
Going deeper
Crawling is the stage where bots start at a URL, follow links and pull the HTML for each page they reach. Indexing and ranking come later — if a page is never crawled, the rest of the pipeline simply does not run.
In day-to-day work, more than half of 'why is this page not showing up?' questions come down to a crawl issue: a blocked robots rule, infinite URL parameters, 5xx errors, JavaScript that never renders. A quick log review usually surfaces the cause within minutes.
The same plumbing applies to GEO. AI crawlers like GPTBot, PerplexityBot and ClaudeBot are still ordinary HTTP bots, so any page that classic search bots cannot crawl will also be hard to cite inside an AI answer.
Related terms
Crawlability
Crawlability is how easily a search engine bot can reach and follow your site's pages — the precondition for indexing.
SEOCrawl Budget
Crawl budget is the amount of resource a search engine is willing to spend crawling a single site over a given period — a critical factor for indexing efficiency on large sites.
SEOIndexing
Indexing is the step where a search engine stores a crawled page in its database. If a page is not indexed, it cannot appear in search results at all.
SEOrobots.txt
robots.txt is the text file at a site's root that tells search engines and AI crawlers which paths they may or may not crawl — a long-standing web standard.
SEOLog File Analysis
Log file analysis is the practice of inspecting raw server access logs to see exactly how search and AI bots crawl your site — the most direct, no-guesswork view of crawl behaviour.
How does your brand show up in AI answers?
Villion measures how your brand appears across ChatGPT, Perplexity and AI Overviews, then automates the work that lifts citation rate and share of voice.
Get a free audit