TL;DR
Why Google crawls thousands of auto-generated pages and refuses to index them — how to read the two Search Console statuses, and what actually fixes it.
→ See how this applies to your business (free 30-min call)Programmatic SEO indexation failure is when Google crawls your auto-generated pages and refuses to index them — Search Console shows most URLs stuck in "Crawled – currently not indexed" or "Discovered – currently not indexed." It is almost never a technical bug. It is a quality verdict. Google's systems compare your 4,000 city-service pages, find that 92% of the text is identical boilerplate with a swapped variable, and decline to spend storage on them. The fix is not more sitemaps or more submissions: it is fewer pages, each with data that exists nowhere else, linked from pages Google already trusts. Below is the diagnostic sequence, the real quotas that constrain your options, and the honest case for not building programmatic pages at all.
The Two Failure States Mean Completely Different Things
People treat these Search Console statuses as interchangeable. They aren't, and confusing them wastes a month.
"Discovered – currently not indexed" means Googlebot knows the URL exists but has not fetched it. This is a *crawl allocation* problem. Google looked at your site's overall value, decided the cost of crawling 8,000 more URLs wasn't justified, and queued them indefinitely. Common on sites with slow server response — if your time to first byte exceeds ~600ms under crawl load, Googlebot throttles itself automatically.
"Crawled – currently not indexed" means Googlebot fetched the page, rendered it, evaluated it, and said no. This is a *quality* verdict. No amount of sitemap resubmission changes it. We have watched clients resubmit the same 3,000 URLs eleven times across four months with a 0% change in index rate.
The practical split:
The Quotas Nobody Reads Until They Hit Them
Before you plan a 20,000-page build, know the hard limits you're working inside:
Submitting a URL is a request for consideration, not a request for indexing. Google has never promised to index anything you submit, and at programmatic scale it usually doesn't.
Why Templates Fail: The 40% Rule
The mechanism is uniqueness ratio. If your template is 900 words and only 120 of them change per page, 87% of the page is identical across your entire set. Google's duplicate detection collapses that cluster and indexes a handful of representatives.
In builds we've run for home service and healthcare clients, the threshold where index rates stop collapsing sits around 40% genuinely unique on-page content — and "unique" means unique *data*, not spun sentences. A page about emergency plumbing in Tulsa needs Tulsa-specific facts: local permit requirements, average response times pulled from your dispatch system, actual jobs completed, neighborhood names your crews serve, real pricing for that market.
Ten client programmatic builds we've tracked since 2023 show the pattern clearly:
The site with 12,000 pages and 4% indexation had spent roughly $26,000 on content generation and hosting. The 480 indexed pages generated 31 organic conversions in a year — about $840 in acquisition cost per conversion, worse than the Google Ads campaign it was supposed to replace.
The Orphan Page Problem Is Bigger Than the Content Problem
Most programmatic sets exist only in the sitemap. Nothing on the site links to them. Google treats sitemap-only URLs as low-priority hints, and crawl depth matters enormously: pages more than three clicks from the homepage get crawled meaningfully less often, and pages with zero internal links get crawled once and forgotten.
What actually moves the needle:
A regional HVAC client we worked with had 1,400 location pages at 11% indexation. We deleted 900 of them, built 32 county-level hubs, and added inline links from 40 existing blog posts. Index rate hit 68% in seven weeks with zero content rewrites on the surviving pages.
When Programmatic SEO Is Not Worth It — Read This Before You Buy
This is the section that costs us business, and it's the most important one here.
Do not build programmatic pages if you serve fewer than 15 distinct locations or offer fewer than 8 distinct services. With a 5×5 matrix you have 25 pages. That's not programmatic SEO, that's a week of writing 25 good pages by hand, and hand-written will outperform templated every time at that scale.
Do not build them if you have no proprietary data. If the only variable you can swap is the city name pulled from a census list, you will produce doorway pages. Google's spam policies name this explicitly, and beyond non-indexation, large doorway sets can trigger a manual action that suppresses your *entire domain* — including the pages that were working. We have seen a site lose 54% of total organic traffic after a doorway-page manual action, taking five months to recover after removing 2,200 pages.
Do not build them if your domain is under a year old with fewer than 20 referring domains. New domains get minimal crawl allocation. Dumping 3,000 URLs on a site with no authority is the fastest route to "Discovered – currently not indexed" across the board. Spend the first 9–12 months on 30 excellent pages and link acquisition.
Do not build them if your search demand doesn't segment. Run the check: if "plumber Tulsa" and "plumber Broken Arrow" return substantially the same SERP with the same intent, Google doesn't want two pages. Query-level differentiation has to exist before page-level differentiation earns anything.
Do not build them if you can't maintain them. A 600-page set needs data refreshes. Stale pricing, closed locations, and dead phone numbers degrade the whole cluster. Budget 4–8 hours monthly for maintenance per 500 pages, indefinitely.
Realistic economics. A properly researched programmatic build — data sourcing, template design, hub architecture, QA — runs $8,000–$25,000 upfront plus $1,500–$4,000/month in maintenance and iteration. Timeline to meaningful organic revenue is 6–11 months, not 6 weeks. If your annual marketing budget is under $30,000, put it into paid acquisition and 20 strong pages instead. Our ROI calculator and pricing pages will tell you the same thing our sales calls do.
Failure modes even when done right:
The 200-Page Pilot That Saves You $20,000
Never build the full set first. Build 200 pages, covering your highest-demand segment, with genuine per-page data. Submit them, then wait 60 days without touching anything.
Read the result honestly:
Pair the pilot with server log analysis. Logs tell you whether Googlebot is even requesting the URLs — Search Console won't distinguish "never fetched" from "fetched and dismissed" fast enough to matter. If Googlebot hit 190 of your 200 pilot URLs and indexed 30, that's a content verdict. If it hit 24 of 200, that's a crawl budget and linking problem, and the content is fine.
If you want a second opinion on whether your page set is salvageable or should be deleted, our free marketing audit includes a crawl-and-index diagnostic, and our services page lays out what a rebuild actually involves. Sometimes the recommendation is to delete 80% of what you built. That's a real outcome, and it's usually the profitable one.
Frequently Asked Questions
What does "Crawled – currently not indexed" mean for programmatic pages?
Google fetched and rendered the page, evaluated it, and chose not to store it. Access is not the problem — the page is technically reachable. The verdict is about value: Google judged the content too similar to other pages or too thin to justify a slot in the index.
How is "Discovered – currently not indexed" different?
Discovered means Google knows the URL exists but has not fetched it yet, usually because crawl scheduling deprioritized your site or the server responded slowly. Crawled means it was fetched and rejected. Discovered points to crawl capacity and internal linking; Crawled points to page quality.
Will submitting a sitemap or requesting indexing fix the problem?
No. Sitemaps help Google discover URLs, not evaluate them, and manual indexing requests are rate-limited and do not override quality assessment. If pages were crawled and rejected, resubmitting the same content produces the same verdict. Change the pages or reduce their number instead.
How many programmatic pages should a site actually publish?
Publish only as many as you have genuinely distinct data to fill. A few hundred pages with unique pricing, inventory, or measured data typically outperform thousands of template clones. If you cannot state what makes a page different from its siblings, that page should not exist.
Free Weekly Briefing
One AI Marketing Tactic.
Every Tuesday. Free.
What's actually working across our client accounts right now — ROAS moves, follow-up sequences, creative angles. The stuff that isn't in any blog post yet.
No spam. Unsubscribe anytime. 1,200+ business owners already in.