AI Crawler

AI Crawler is the umbrella label for crawlers operated to train LLMs or to power AI search answers. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot all sit under it — as the list keeps growing, policy docs and marketing decks needed a single phrase to talk about them.

The two decisions inside an 'AI crawler policy' are usually: (1) is our content allowed into LLM training data, and (2) do we want to be eligible for citation inside AI answers. Those answers can diverge, which is why more sites configure rules per bot rather than blanket-allow or blanket-block.

The fleet keeps expanding, so a robots.txt set once and forgotten will quietly miss newly launched bots. The standard pattern is to refresh the bot list periodically and apply the same policy intent — opt out of training, allow answer citation, whatever it is — consistently across the family.