Crawl and index controls
Crawl budget
Crawl budget is the time and resources Google can devote to crawling a single site.
In plain words
The web is nearly endless, and Google can’t crawl every URL all the time. So each site gets a limited share of its attention.
For most sites, this never becomes a problem.
Example
Google’s crawl budget guide is meant mainly for large sites with over 1 million unique pages that change about weekly, and for sites with over 10,000 pages that change daily.
Why it matters for AI search
Google says its generative AI features use publicly accessible, crawlable content, so crawlability matters for AI search too.
For very large and frequently updated sites, Google points to its crawl budget guide.
What you can do
- Running a small site? Focus on content and internal links, not on crawl budget.
- Running a large site? Cut duplicate and useless URLs, such as endless filter combinations.
- Keep your sitemap up to date and your server fast.
- If Googlebot puts too much load on your server, Google explains how to reduce its crawl rate.
Often confused with
- Indexing
- Crawling means fetching a page; indexing means storing it for search. A crawled page is not always indexed.
Sources
- Google: Large site owner's guide to managing your crawl budget
- Google Search Central: Optimizing your website for generative AI features on Google Search
Facts and sources last checked on 19 October 2026.
Related terms
How does your page do in AI search?
GEO-Score checks one page on 22 signals, from AI crawler access to structured data, and shows what to fix first.
Analyze a page