Large-site SEO: indexed, crawled, and kept.

Discovery is rarely the real problem

When a large site has pages that are not indexed, the instinct is to conclude that search engines do not know about them, and to go looking for a way to tell them harder. This is almost always the wrong diagnosis.

Search engines are good at discovery. If a URL is in a sitemap that is being fetched, and especially if it is linked internally, they know it exists. Non-indexation after that point is a judgement, not an oversight — the page was seen and not considered worth storing, or it was seen and considered a duplicate of something already stored, or it could not be fetched cleanly.

The useful reframing: discovery, crawling, and indexing are three separate stages with three separate failure modes, and treating them as one problem produces the wrong fix roughly two times in three.

Before doing anything else, verify the page itself. Fetch it and check the status code, the robots meta tag, the canonical, and what is actually in the rendered DOM. A meaningful share of "not indexed" findings turn out to be a page returning a redirect, carrying a noindex from an unlaunched-market gate, or rendering empty. Only the case that returns 200 with an indexable directive and real content is a discovery problem at all.

The discovery ladder, in order

For a site of any size, there is a sensible order of operations. It has changed recently, and a lot of published advice has not caught up.

1. Sitemaps — the default for everything. Every indexable URL belongs in a sitemap, and the sitemap needs to be complete, current, and fetchable. A page missing from the sitemap is the bug worth fixing first, and it is a surprisingly common one on sites where the sitemap is generated from a different query than the routing layer uses.
2. Internal links — the part people skip. The sitemap gives discovery; links give crawl priority and pass equity. A page that appears only in a sitemap is discoverable but unimportant, and on a large site "unimportant" resolves to "not indexed" quite often. If a page cannot be reached by clicking from somewhere, ask whether it should exist.
3. IndexNow for the engines that accept it. Useful, cheap, and worth wiring up for Bing and the other participating engines. Two practical notes: it does not feed Google, and the key is per-host — the same key across multiple hostnames returns errors rather than working quietly.

The uncomfortable implication is that there is no lever to pull. If a page is in the sitemap, linked internally, fast, and distinct, and it is still not indexed, the answer is that it was not judged worth indexing — and the response is to make it worth indexing or to stop publishing that kind of page.

Crawl budget is mostly a response-time problem

Crawl budget is discussed as though it were an allocation handed down by the search engine. It behaves much more like a consequence of how the site performs.

The pattern we observe consistently across large properties: crawl rate tracks server response time closely. When responses slow — a heavy query, a cold cache, an expensive rebuild — crawl rate falls, and it falls quickly. When responses recover, crawl rate climbs back. The engine is adjusting to avoid degrading the site.

That makes performance work an indexing intervention, not just a user-experience one. Concretely:

• Watch time-to-first-byte for crawler traffic specifically, not just aggregate field data. Crawlers frequently hit cold paths that real users, served from cache, never touch — so a site with excellent user-facing metrics can still be serving crawlers slowly.
• Cold rebuilds are a crawl event. If a large generated resource — a sitemap index, a heavy listing page — takes tens of seconds to rebuild after invalidation, crawlers will hit that window. Pre-warm or serve stale rather than making the crawler wait.
• Check what proportion of crawl is being spent usefully. Faceted parameters, session artefacts, paginated depth, and internal search results routinely consume a large share of fetches on sites that have never audited it. Removing worthless URLs increases the budget available to real ones without changing anything about the server.
• Cached responses may be invisible in origin logs. If a CDN serves the crawler, your origin never sees the request, and a log-based crawl analysis will understate real activity — sometimes dramatically. Use the CDN's own analytics as well.

What deserves a URL

The instinct on a data-driven site is to generate a page for every combination the data supports. Every city, every category, every intersection of the two. It is easy, it looks like coverage, and it is the most reliable way we know to produce an indexing problem.

Governance means deciding, in code, which records earn a page. It is a rule applied at publish time, not a cleanup performed later. Criteria that tend to work:

• Is there enough distinct content? If two pages differ only by a place name substituted into the same sentences, they are one page as far as an index is concerned, and publishing both dilutes whichever one might have ranked.
• Is there real demand? Not keyword-tool volume — evidence that anyone is looking for this specific intersection. A locale or category superset generated because the data model allowed it is not demand.
• Does it have somewhere to sit? If nothing links to it naturally, it is a page you will have to prop up permanently.
• Is it durable? Content that will be obsolete in three weeks and left behind is a maintenance liability with a URL attached.

The counterintuitive result is that publishing fewer pages often increases total indexed pages, because the ones that remain are distinct enough to be worth storing and concentrated enough to attract links and crawl attention. Consolidation is an underused lever and an unpopular one, because the reporting shows a reduction in published pages before it shows the improvement.

Near-duplicates: the problem underneath most indexing problems

On large generated sites, the most common indexing failure is duplication, and it is usually not obvious duplication. It is dozens of pages that are meaningfully the same thing appearing under different URLs because the data model produced them separately.

Recurring items are the classic case. A weekly event, ingested from a feed with one record per date, becomes fifty-two URLs describing the same thing. Each is thin, each competes with the others, and collectively they consume crawl budget that the rest of the site needed. The fix is to collapse them into one evergreen page with the dates as content rather than as URLs — a decision that belongs in the ingest layer, because after publication you are cleaning up rather than preventing.

The same shape shows up in other forms: the same item arriving from two feeds under slightly different names; a product listed under multiple categories with a URL for each; a location addressed by both its administrative name and its colloquial one.

Where duplicates already exist and have accumulated history, the sequence that works is: pick a canonical, redirect the twins to it permanently, keep a record of what was collapsed so the ingest layer does not recreate them, and purge any CDN cache holding the old responses — otherwise the redirect is live at the origin and invisible at the edge for as long as the cache holds. Prioritise by traffic rather than by count; a handful of duplicated pages with real traffic matter more than a long tail with none.

Plan for pages that die

Every large site has content that expires: events that finish, products that delist, listings that get withdrawn. What happens to those URLs is a design decision, and if it is not made deliberately it defaults to the worst option — a 404 for a URL that has links, history, and possibly traffic.

A defensible policy has a few layers:

1. Never let an unpublish become an accidental 404. When a record is demoted or withdrawn, the route should still resolve to something useful — a successor, a parent, or a page that explains the state — rather than disappearing.
2. Keep tombstones. A record of URLs that once existed and what they should now resolve to. Without this, a later data refresh has no way of knowing that a path was ever live.
3. Redirect to the nearest genuinely relevant thing. A finished event should point at the recurring series or the venue, not at the homepage. Mass redirects to the root are treated as soft 404s and help nobody.
4. Make the resolver handle sub-routes. If a page has children, they need the same treatment, and they are usually forgotten.

The reason to design this early is that retrofitting it requires reconstructing which URLs used to exist — from logs, from search-console data, from backups — which is considerably more work than recording it at the time.

Multilingual: where large sites break quietly

Adding languages multiplies every problem above, and introduces a few of its own. The failures are quiet — nothing errors, the pages look correct, and the wrong version ranks in the wrong market for months.

• hreflang must be reciprocal and self-referential. Every page in a cluster lists every other page and itself. A cluster missing the self-reference is commonly ignored wholesale, and nothing in the page indicates a problem.
• Unprefixed URLs resolve to the default language. Whatever your routing does with a path that carries no locale, know what it is, because it determines which version accumulates authority — and a redirect rule that sends users to a guessed locale will send crawlers there too.
• Do not translate into a locale with no demand. A superset of locales generated because the framework supports them produces thin duplicates across languages, which is the duplication problem again with a translation budget attached.
• Translation stubs should be upgraded, never deleted. Removing a partial translation removes a URL that may already be indexed and linked. Improve it in place.
• Locale trees need locale-appropriate content, not just locale-appropriate strings. A market page translated from another market's page, with the operators, payment methods, or regulatory context of the source market, is worse than not having the page.

What to watch, and how often

Large sites degrade gradually, so the monitoring matters more than the audit. A weekly view that catches direction is worth more than an exhaustive quarterly report.

• Indexed count against published count, as a ratio and as a trend. The absolute numbers matter less than the gap widening.
• Crawl requests and average response time, together. They move in opposite directions, and seeing them on one chart makes the relationship obvious.
• Sitemap completeness as an automated check. Compare the routing layer's list of indexable URLs against the sitemap's contents and alert on divergence. This catches the single most common structural bug.
• New URLs that received no impressions after a fixed window. A far better early-warning signal than waiting for a traffic decline.
• 404s and redirect chains, from real crawl data. Particularly after any bulk data operation, which is when they appear.