Crawl Budget
In one line
Crawl budget is the amount of resource a search engine is willing to spend crawling a single site over a given period — a critical factor for indexing efficiency on large sites.
Going deeper
Crawl budget is roughly the product of two factors: 'crawl rate limit' (driven by your server speed and stability) and 'crawl demand' (driven by how valuable Google thinks your content is). Fast, fresh sites earn larger budgets naturally.
If you have a few thousand pages, you can mostly ignore crawl budget. It becomes critical for ecommerce, publishers and UGC platforms with hundreds of thousands of URLs, where parameter combinations, pagination and infinite filters quietly waste it.
The optimisation playbook is 'plug the leaks, push crawl toward what matters'. Block infinite parameter combos in robots.txt; align canonicals, sitemaps and internal links so the crawler concentrates on core pages. Improving CWV also lifts the budget indirectly by speeding up responses.
Sources
Related terms
robots.txt
robots.txt is the text file at a site's root that tells search engines and AI crawlers which paths they may or may not crawl — a long-standing web standard.
SEOSitemap.xml
An XML sitemap is a structured file that lists the important URLs of a site so search engines can discover and crawl them efficiently — its impact grows with site size.
SEOCanonical Tag
The canonical tag tells search engines which URL is the master version when multiple URLs serve the same or near-duplicate content.
SEOCore Web Vitals (CWV)
Core Web Vitals (CWV) is Google's set of headline metrics for quantifying page user experience — covering loading, interactivity and visual stability.
SEOInternal Linking
Internal linking is the practice of connecting pages within the same domain — and it shapes crawl efficiency, how authority flows between pages, and how users move through the site.
How does your brand show up in AI answers?
Villion measures how your brand appears across ChatGPT, Perplexity and AI Overviews, then automates the work that lifts citation rate and share of voice.
Get a free audit