AI Crawling
In one line
AI Crawling refers to the act of AI training or answer-generation bots traversing the web — distinct from classic search crawling in both policy expectations and load profile.
Going deeper
AI Crawling is the act of AI crawlers — GPTBot, ClaudeBot, Google-Extended and the rest — actually traversing the web. It is easy to lump in with classic search crawling, but two things differ: the policy decisions split (training vs. answer-citation), and some AI bots run a heavier load, which has operational implications.
Why it helps marketers to understand the act, not just the bot list: it makes robots.txt decisions stop looking binary. 'Opt out of training but stay eligible for citation' only makes sense once you can see which bot is crawling for which purpose.
Worth flagging: AI crawling is increasingly happening live, at the moment the user requests an answer. Agents like ChatGPT-User fetch pages on demand, with user-agent strings that change more often than training bots and policies that differ from them. That part of the fleet rewards regular review.
Related terms
AI Crawler
AI Crawler is the umbrella term for web crawlers operated for AI model training or AI-search answer generation — GPTBot, ClaudeBot and Google-Extended are the canonical examples.
GEO·AEOGPTBot
GPTBot is OpenAI's official web crawler used for ChatGPT training and search indexing — controllable via robots.txt.
GEO·AEOOAI-SearchBot
OAI-SearchBot is OpenAI's separate crawler for ChatGPT Search indexing — distinct from GPTBot, so you can control training and search-indexing access independently.
GEO·AEOClaudeBot
ClaudeBot is Anthropic's web crawler used for training Claude and grounding its answers — manageable via robots.txt.
GEO·AEOllms.txt
llms.txt is a proposed text file placed at the site root that tells large language models where the most important content lives — think 'sitemap, but written for LLMs'.
How does your brand show up in AI answers?
Villion measures how your brand appears across ChatGPT, Perplexity and AI Overviews, then automates the work that lifts citation rate and share of voice.
Get a free audit