Crawl budget

Why it matters

On a very large site, thousands of pages, an inefficient crawl budget means some pages, potentially valuable ones, never get crawled and indexed at all, simply because the search engine ran out of allotted crawl activity before reaching them. This matters far less on a small site with a handful of pages, but becomes a real constraint at scale. It also interacts with site speed directly: a server that responds slowly effectively shrinks its own crawl budget, since search engines throttle how fast they crawl a site specifically to avoid overwhelming a server that is already struggling. It also interacts with the XML sitemap directly: a sitemap that accurately reflects only the pages actually worth crawling helps direct limited crawl activity toward them rather than leaving a search engine to discover everything on its own.

What goes wrong when it's handled badly

A large site with many low-value, near-duplicate pages, thin filtered category variations, for example, can waste crawl budget on pages that will never meaningfully rank, starving genuinely valuable pages of crawl attention. Pruning or consolidating thin, near-duplicate pages is one of the more effective ways to redirect crawl budget toward what actually matters. A concrete example is an ecommerce site generating thousands of unique, indexable URLs purely from filter and sort combinations on the same underlying product set, spending the bulk of its crawl budget on pages that are functionally duplicates of each other. Blocking low-value, parameter-driven URLs from crawling explicitly, rather than leaving every possible combination technically reachable, is the standard fix once this pattern is identified. Checking Search Console's own crawl statistics periodically is the direct way to see whether crawl activity is actually reaching the pages that matter most.