Ecommerce Indexing with the Indexing API

This guide is for store owners and developers who manage large product catalogs and need steady ecommerce indexing without guesswork. Product pages often sit in Discovered or Crawled states for weeks because facets, variants and thin descriptions dilute crawl attention. You will learn how crawl budget, sitemaps, internal links and the Google Indexing API fit together, what the API supports for JobPosting and BroadcastEvent content, and how to handle product URLs honestly within those limits. By the end you can audit coverage, clean sitemaps, prioritize submissions and automate daily updates in Python with logging and retry handling.

Key takeaways

Product pages need clean sitemaps, stable canonicals and strong category links before any API hint helps.
The Google Indexing API documents JobPosting and BroadcastEvent only, so use it carefully for products and rely on sitemaps and internal links for broad coverage.
Google does not support IndexNow, so send product updates to IndexNow for Bing side engines and use Google paths separately.
Track coverage by SKU, log every submission with status codes, and run a weekly routine to remove 404s and noindex URLs from queues.

Why product pages stay unindexed for weeks
How crawl budget works on large catalogs
What the Google Indexing API can and cannot do for products
Building a product sitemap that Google actually reads
Using URL_UPDATED and URL_DELETED for inventory changes
Internal linking patterns that move new products into the index
Handling variants, out of stock pages and discontinued products
Measuring index coverage for thousands of SKUs
Automating submissions with Python for daily catalog updates
Common ecommerce indexing mistakes that waste quota
A weekly routine to keep product pages indexed

Ecommerce indexing cover showing product grid and indexing flow with mint highlights

Why product pages stay unindexed for weeks

This section covers why product pages stay unindexed for weeks in the context of ecommerce indexing. For stores with thousands of SKUs, small crawl traps compound quickly, so start with data on which URLs Google has seen and which remain undiscovered. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Google discovers most pages through crawl, not through a single submission. A submission is a hint that asks for a fresh look, but ranking and storage still depend on quality, uniqueness and site trust. That is why steady technical hygiene matters more than any one push. Keep response times low, avoid redirect chains, and return clear status codes. When the crawler can fetch quickly and without loops, each hint carries more weight and uses less of your daily allowance.

Sitemaps remain the backbone of discovery. A clean product or article sitemap lists only canonical, indexable URLs that return 200 and load quickly. Split large catalogs into chunks of 10,000 to 40,000 URLs, compress with gzip, and reference each chunk from a sitemap index. Update the lastmod field only when content truly changes. Submit the index in Search Console and keep it reachable. A tidy sitemap reduces wasted fetches and leaves room for priority pages.

Crawl budget is often misunderstood. For small sites it rarely limits indexing, but for catalogs with 50,000 to 500,000 URLs it shapes what gets visited each day. Facets, session parameters, internal search results and duplicate variants can trap crawlers in low value loops. Use robots rules to block filtered views, use canonical tags to consolidate variants, and link best sellers from the home page and category hubs. Fewer dead ends means faster visits to new and updated pages.

In practice, pull your product URL list from the commerce platform, join it with Search Console coverage data, and flag items that are new, changed in price or availability, or stuck as Discovered without indexing. Remove URLs that return 404, carry noindex, or canonical to another page. Sort the remainder by margin, stock depth and search demand. Submit only the top slice through paced jobs, and let sitemaps plus internal links carry the rest. This filter often cuts submit volume by half while lifting index share for sellable items.

Check	What to look for	Action
Sitemap URLs	200, canonical, indexable	Remove 404, noindex and non canonical entries
Lastmod	True change date	Update only on real content change
Canonical	Self referencing on main	Point variants to preferred URL
Internal links	Clicks from home	Add new products to category hubs
Availability	In stock markup	Keep out of stock pages clear and crawlable

How crawl budget works on large catalogs

This section covers how crawl budget works on large catalogs in the context of ecommerce indexing. Catalog size, facet depth and server speed decide how many product URLs Google can revisit each day, so measure before you submit. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Search Console verification is the gate for any Google submission. The service account that calls the API must be added as an Owner on the exact property, including the correct scheme and subdomain. Domain properties and URL prefix properties behave differently, so match the property you verify with the URLs you submit. If you see permission denied or 403, check sharing settings first, then OAuth scope, then key expiry. Most auth failures trace to a missed sharing step, not to code.

The Google Indexing API only documents JobPosting and BroadcastEvent pages, which covers job listings and livestream video. Many site owners still test it for product or article URLs, but that use is off label and results vary. Google may process the hint, ignore it, or throttle it. State this plainly to stakeholders. Use the API for eligible content first, and rely on sitemaps, internal links and IndexNow for broad coverage on other page types.

Google does not support IndexNow, so plan for two ecosystems. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for eligible types. A practical setup sends product updates to both paths at publish time. One worker prepares the URL list, then one branch pings IndexNow endpoints and another branch queues Google notifications within quota. Coverage improves without double counting.

For background on a related setup, see how news publishers structure fast indexing which explains how publishers structure notifications and sitemaps for time sensitive pages.

Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

Diagram showing ecommerce indexing flow with crawl, sitemap and queue steps

What the Google Indexing API can and cannot do for products

This section covers what the google indexing api can and cannot do for products in the context of ecommerce indexing. The API accepts URL_UPDATED and URL_DELETED for eligible types, which means product use needs careful notes and fallback paths. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Quotas shape every automation decision. Many projects start with about 200 publish requests per day for URL notifications, plus per minute limits that trigger 429 when bursts arrive. Track usage in Cloud Console under APIs and Services, set alerts at 60 percent and 85 percent, and log each publish with timestamp, URL, response code and notification type. When you know your burn rate by hour, you can pace jobs, defer low priority URLs and avoid midnight surprises.

A 429 means slow down, not try harder. Read the Retry After header when present, then wait with exponential backoff and jitter before retrying. A common pattern waits 2 seconds, then 4, then 8, then 16, with a small random addition to avoid synchronized retries. Cap retries at 4 or 5 and move the URL to a delayed queue after that. Hammering the endpoint during a limit only extends the block and burns log space.

A 403 usually points to permissions or scope. Confirm the service account email has Owner access in Search Console, confirm the OAuth scope includes the indexing scope, and confirm the JSON key file matches the active key in Cloud Console. Check clock skew on the server, since JWT auth fails when time drifts by more than a few minutes. Rotate keys on a schedule, store them in a secret manager, and never paste private keys into chat tools or shared docs.

For official details, see sitemap protocol details which documents the expected fields and behavior.

Check	What to look for	Action
Sitemap URLs	200, canonical, indexable	Remove 404, noindex and non canonical entries
Lastmod	True change date	Update only on real content change
Canonical	Self referencing on main	Point variants to preferred URL
Internal links	Clicks from home	Add new products to category hubs
Availability	In stock markup	Keep out of stock pages clear and crawlable

Building a product sitemap that Google actually reads

This section covers building a product sitemap that google actually reads in the context of ecommerce indexing. A sitemap that lists only canonical 200 URLs with accurate lastmod helps Google spend fetches on sellable pages. Good product sitemap indexing also means one sitemap per store section with clean lastmod, which keeps large site indexing stable as the catalog grows past 100,000 URLs. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

JWT failures look cryptic but follow a pattern. Invalid signature often means the wrong key file or a corrupted newline in the private key. Invalid grant often means the service account is disabled or the project never enabled the API. Failed to parse often means the token was truncated in logs or copied with extra spaces. Keep token lifetimes short, request a fresh access token per batch, and log the key ID without logging the secret. Small hygiene steps remove most auth noise.

Logging turns guesses into fixes. For each submission store the URL, notification type, HTTP status, response body snippet, latency and a correlation ID. Keep success and error logs separate so you can scan error rates by hour. Export daily counts to a sheet or dashboard that shows submits, 200 responses, 403 responses, 429 responses and remaining quota. When stakeholders ask why a product is not visible, you can point to exact evidence instead of general theories.

Internal linking does more for indexing than most teams expect. New URLs that sit four clicks from the home page may wait days for a visit, while URLs linked from a popular category or a recent posts block get visited quickly. Add new products to relevant category pages, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple, stable links help both Google and IndexNow driven crawlers find changes fast.

To compare paths side by side, read alternatives that cover Bing and other engines before you commit quota to one route.

Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

Using URL_UPDATED and URL_DELETED for inventory changes

This section covers using url_updated and url_deleted for inventory changes in the context of ecommerce indexing. Inventory churn creates constant add, update and remove signals, so map each stock event to the correct notification type. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Canonical tags decide which URL keeps the indexing credit. If variants with color, size or tracking parameters lack a canonical, Google may pick a different URL or delay indexing while it compares duplicates. Point each variant to the preferred canonical, keep the canonical self referencing on the main URL, and make sure sitemaps list only canonicals. For translated or regional pages, add hreflang and keep each locale self consistent. Clean signals shorten the decision time.

Robots directives and meta tags can silently block indexing. A stray noindex in a template, an X-Robots-Tag header from a staging config, or a disallow in robots that covers new paths will keep pages out even after successful submission. Audit headers with a fetch tool, render pages as Googlebot, and check the coverage report for Excluded by noindex or Blocked by robots. Fix the template once rather than patching URLs one by one.

Thin or duplicated content slows indexing because Google prioritizes pages likely to satisfy searchers. Short product descriptions copied from suppliers, empty category pages and near duplicate articles often sit in Discovered or Crawled without indexing. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing.

Check	What to look for	Action
Sitemap URLs	200, canonical, indexable	Remove 404, noindex and non canonical entries
Lastmod	True change date	Update only on real content change
Canonical	Self referencing on main	Point variants to preferred URL
Internal links	Clicks from home	Add new products to category hubs
Availability	In stock markup	Keep out of stock pages clear and crawlable

# Python submit with backoff and logging, no em dash in comments
import time, random, requests
from google.oauth2 import service_account
from google.auth.transport.requests import Request

SCOPES = ['https://www.googleapis.com/auth/indexing']
creds = service_account.Credentials.from_service_account_file('key.json', scopes=SCOPES)
creds.refresh(Request())
headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer ' + creds.token}
payload = {'url': 'https://example.com/jobs/123', 'type': 'URL_UPDATED'}
for attempt in range(5):
    r = requests.post('https://indexing.googleapis.com/v3/urlNotifications:publish', json=payload, headers=headers, timeout=15)
    print(r.status_code, r.text[:300])
    if r.status_code != 429:
        break
    wait = 2 ** attempt + random.uniform(0, 1)
    time.sleep(wait)

Internal linking patterns that move new products into the index

This section covers internal linking patterns that move new products into the index in the context of ecommerce indexing. Category hubs and related product blocks pass crawl attention to new items faster than isolated detail pages. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Response codes guide the next action. A 200 with URL_UPDATED means the hint was accepted, not that the page is indexed. A 200 with URL_DELETED means the removal hint was accepted. A 400 means the request body was malformed. A 401 means auth failed. A 403 means permission failed. A 404 on metadata means no notification history exists. A 429 means quota or rate pressure. Map each code to a runbook entry so on call staff know whether to retry, fix auth or pause.

Bulk work needs a queue, not a loop without pauses. Push new and updated URLs into a table or message queue with fields for URL, change type, priority, attempts and next retry time. A worker pulls 1 to 5 URLs per minute during normal hours and slows further when 429 appears. High priority URLs such as new jobs, live videos or price changes go first, while low priority refreshes wait. This pacing respects quotas and keeps logs readable.

Edge automation can speed discovery for Bing side engines. Cloudflare Workers or a deploy hook can fire an IndexNow ping the moment a page publishes, without waiting for a nightly job. Store the IndexNow key in a secret, build the JSON payload with host, key and URL list, and POST to the IndexNow endpoint with a timeout of 10 seconds. Log response codes 200, 202, 400, 403 and 422 separately. Fast pings plus clean sitemaps give broad engines a clear trail to follow.

A useful companion is when to send URL_UPDATED or URL_DELETED which clarifies when each notification type is appropriate.

Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

Workflow for ecommerce indexing showing publish, log and retry stages

Handling variants, out of stock pages and discontinued products

This section covers handling variants, out of stock pages and discontinued products in the context of ecommerce indexing. Variants and out of stock states need clear canonical and availability signals to avoid duplicate or thin page traps. A clear out of stock indexing rule keeps returning products crawlable while discontinued URLs leave the index, and strong category page indexing carries new arrivals when variant URLs are consolidated. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Monitoring in Cloud Console prevents most outages. Open the project, select the Indexing API, and review traffic by response code, median latency and errors over 7 and 28 days. Create an alert that fires when error rate passes 5 percent or when daily publishes pass 80 percent of quota. Review which service account or key generates the most traffic. If a test script runs wild, you will see it within minutes and can disable the key before quota is gone.

Recovery after exhaustion is about order, not speed. Pause new publishes, export the pending list, remove duplicates and noindex URLs, then sort by business priority and age. Resume at a low rate such as 1 request every 30 to 60 seconds and watch for 429. If errors return, halve the rate and extend the pause. Most daily quotas reset around midnight Pacific, but per minute throttles clear sooner. Document the incident so the next launch uses a safer schedule.

Security for service account keys deserves steady attention. Create one project per environment, grant only the roles needed for publishing, and store JSON keys in a vault with rotation every 60 to 90 days. Restrict key use by IP or workload where the platform allows it. Delete old keys after rotation and audit who accessed the vault. If a key leaks in a repo or log, revoke it at once, create a replacement, and review submissions made with the exposed key.

Check	What to look for	Action
Sitemap URLs	200, canonical, indexable	Remove 404, noindex and non canonical entries
Lastmod	True change date	Update only on real content change
Canonical	Self referencing on main	Point variants to preferred URL
Internal links	Clicks from home	Add new products to category hubs
Availability	In stock markup	Keep out of stock pages clear and crawlable

// Node submit with retry, plain logging
const { JWT } = require('google-auth-library');
async function publish(url, type) {
  const auth = new JWT({ keyFile: 'key.json', scopes: ['https://www.googleapis.com/auth/indexing'] });
  const token = await auth.getAccessToken();
  for (let i = 0; i < 5; i++) {
    const r = await fetch('https://indexing.googleapis.com/v3/urlNotifications:publish', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', Authorization: 'Bearer ' + token.token },
      body: JSON.stringify({ url, type })
    });
    console.log(r.status, await r.text().then(t => t.slice(0, 300)));
    if (r.status !== 429) break;
    await new Promise(x => setTimeout(x, 1000 * (2 ** i) + Math.random() * 500));
  }
}

Measuring index coverage for thousands of SKUs

This section covers measuring index coverage for thousands of skus in the context of ecommerce indexing. Coverage reports by SKU reveal patterns such as entire brands or suppliers stuck outside the index. Track ecommerce index speed as the median days from publish to Indexed for priority groups, and review ecommerce seo crawl signals such as crawl rate, response time and canonical choice alongside coverage. For catalogs approaching million pages indexing, sample by template before judging the whole store. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

If limits shape your plan, review how quota limits shape bulk submissions for quota numbers and pacing examples.

Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

Automating submissions with Python for daily catalog updates

This section covers automating submissions with python for daily catalog updates in the context of ecommerce indexing. Daily feeds from commerce platforms make automation practical when paired with deduplication and priority flags. Tests that send indexing api products through the endpoint often see throttling, so an indexing api store integration should keep volumes low with strict logging. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Protocol specifics are covered in IndexNow documentation if you need to confirm payload format or key handling.

Check	What to look for	Action
Sitemap URLs	200, canonical, indexable	Remove 404, noindex and non canonical entries
Lastmod	True change date	Update only on real content change
Canonical	Self referencing on main	Point variants to preferred URL
Internal links	Clicks from home	Add new products to category hubs
Availability	In stock markup	Keep out of stock pages clear and crawlable

---
// Astro sitemap and feed wiring, runs at build time
import { getCollection } from 'astro:content';
const posts = await getCollection('blog');
const urls = posts.filter(p => !p.data.draft).map(p => ({ loc: '/blog/' + p.slug + '/', lastmod: p.data.updatedDate }));
---
<rss version="2.0">
  {urls.slice(0, 50).map(u => <item><link>{u.loc}</link></item>)}
</rss>

Common ecommerce indexing mistakes that waste quota

This section covers common ecommerce indexing mistakes that waste quota in the context of ecommerce indexing. Quota is small relative to catalog size, so every wasted submit on 404 or noindex URLs delays a sellable page. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

A weekly routine to keep product pages indexed

This section covers a weekly routine to keep product pages indexed in the context of ecommerce indexing. A short weekly review keeps sitemaps, links and queues aligned with what shoppers can actually buy. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Check	What to look for	Action
Sitemap URLs	200, canonical, indexable	Remove 404, noindex and non canonical entries
Lastmod	True change date	Update only on real content change
Canonical	Self referencing on main	Point variants to preferred URL
Internal links	Clicks from home	Add new products to category hubs
Availability	In stock markup	Keep out of stock pages clear and crawlable

FAQ

How long does product indexing usually take?

Most new product pages appear within days to weeks when sitemaps, links and server responses are clean. Large catalogs with facets or thin descriptions often wait longer because crawlers spend time on low value variants. To benchmark ecommerce index speed, record publish date per SKU and compare against Indexed date in coverage data. Clean the sitemap, link new items from category hubs, and monitor coverage weekly. Use paced submissions for priority items and let steady crawl carry the rest.

Can the Indexing API index all my products?

No. The API documents JobPosting and BroadcastEvent pages, which covers job listings and livestreams. Product pages fall outside that documented scope, so results vary and throttling is common. Treat any product test as experimental, keep volumes low, and rely on sitemaps, internal links and IndexNow for broad product coverage.

Does IndexNow help Google product indexing?

IndexNow does not send to Google, since Google does not support the protocol. It notifies Bing, Yandex, Naver, Seznam and other partners. That still matters for stores with Bing traffic or regional reach. Send product updates to IndexNow for those engines and use Google sitemaps plus inspection paths separately.

Should out of stock pages stay indexed?

Keep out of stock pages indexable when the product will return, with clear availability markup and related alternatives. A simple out of stock indexing policy is to keep returning items indexable for 90 days, then reassess demand before removing them. Remove or redirect pages for discontinued items that will never return, then clean them from sitemaps and queues. This balance preserves ranking potential while avoiding thin, frustrating results.

Why do variants cause indexing delays?

Color, size and tracking variants create near duplicates that force Google to choose a canonical. Without clear canonical tags, the choice takes time and the wrong URL may win. Point variants to the preferred canonical, list only canonicals in sitemaps, and block filtered views that add no value.

How do I measure progress across SKUs?

Join your SKU list with Search Console coverage and crawl stats by template, brand and supplier. For product page indexing, track counts for Indexed, Discovered without indexing, Crawled without indexing, and Excluded by reason. Review weekly after sitemap or linking changes. Movement from Discovered to Indexed for priority groups shows the work is effective.

Sources

https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
https://www.indexnow.org/documentation
https://developers.google.com/search/docs/crawling-indexing/request-indexing

Ecommerce Indexing: Speed Up Product Page Indexing with the Indexing API