Googlebot
In one line
Googlebot is Google's official web crawler used to build the search index — controllable through robots.txt rules and identifiable by its User-Agent string.
Going deeper
Googlebot is the crawler that feeds Google's search index. Since mobile-first indexing became the default, Googlebot Smartphone is the primary crawler and the desktop variant plays a supporting role.
Two recurring checks in practice: whether robots.txt accidentally blocks paths you actually want indexed, and whether JavaScript-rendered content is visible to Googlebot. The fastest way to test the second is the URL Inspection tool in Google Search Console.
Google runs other crawlers alongside Googlebot — Google-Extended, Googlebot-News, AdsBot and so on. If you want to separate decisions like 'allow search but opt out of AI training', you need to write rules against the right identifier.
Sources
Related terms
Technical SEO
Technical SEO is the infrastructure-level work — crawling, indexing, rendering, speed, structured data — that makes a site readable and rankable for search engines in the first place.
GEO·AEOGoogle-Extended
Google-Extended is the separate user agent Google uses for training Gemini and Vertex AI, letting site owners control AI training access independently from regular search indexing.
SEOHTTPS
HTTPS (HyperText Transfer Protocol Secure) is HTTP traffic encrypted with TLS — and a baseline ranking signal Google has confirmed publicly.
GEO·AEOGPTBot
GPTBot is OpenAI's official web crawler used for ChatGPT training and search indexing — controllable via robots.txt.
SEOSchema.org
Schema.org is the shared vocabulary co-sponsored by Google, Microsoft, Yahoo and Yandex that lets you label what each page means so search engines and AI can understand it.
How does your brand show up in AI answers?
Villion measures how your brand appears across ChatGPT, Perplexity and AI Overviews, then automates the work that lifts citation rate and share of voice.
Get a free audit