HTTP 429 from the Google Indexing API: Causes, Backoff, and Queues

This guide is for site owners, developers and SEO leads who work with indexing api 429 error and need a reliable routine without guesswork. Many teams hit the same wall: submissions return mixed status codes, logs are thin, and coverage reports move slowly while stakeholders ask for dates. The facts matter here. The Google Indexing API documents JobPosting and BroadcastEvent pages, and Google does not support IndexNow, so every workflow must respect those limits. You will learn exact checks, safe pacing, logging that proves what happened, and recovery steps that work under quota. Follow the sections in order, test with a handful of URLs first, then scale to hourly batches once responses stay clean.

Key takeaways#

• What 429 means on this API starts with identity and access checks, not with faster retries.
• Daily quota versus per minute throttle works best with one test URL and full logs before bulk batches.
• Daily quota and per minute throttling need different responses, track both in Cloud Console.
• Deduplicate, filter noindex and non canonical URLs, then queue at a fixed pace with backoff.

• What 429 means on this API
• Daily quota versus per minute throttle
• Reading the Retry After header
• Exponential backoff with jitter for indexing api 429 error
• Queue design that prevents bursts
• Pacing bulk jobs over hours
• Logging and alerting for throttle events
• Recovery after a long block
• When to defer low priority URLs
• Code for resilient submission workers
• A checklist to stay under the ceiling
• FAQ
• Sources
• Further reading

What 429 means on this API#

This section covers what 429 means on this api in the context of indexing api 429 error. HTTP 429 signals that you sent requests faster than your allowance permits, not that your URLs are invalid. The body often names quotaExceeded or rateLimitExceeded and may include a Retry After hint. Treat it as a pacing instruction: pause, preserve the backlog, and resume slower. Continuing at the same rate extends the block and wastes log space. A calm, logged pause is the fastest path back to steady 200 responses. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Google discovers most pages through crawl, not through a single submission. A submission is a hint that asks for a fresh look, but ranking and storage still depend on quality, uniqueness and site trust. That is why steady technical hygiene matters more than any one push. Keep response times low, avoid redirect chains, and return clear status codes. When the crawler can fetch quickly and without loops, each hint carries more weight and uses less of your daily allowance.

Search Console verification is the gate for any Google submission. The service account that calls the API must be added as an Owner on the exact property, including the correct scheme and subdomain. Domain properties and URL prefix properties behave differently, so match the property you verify with the URLs you submit. If you see permission denied or 403, check sharing settings first, then OAuth scope, then key expiry. Most auth failures trace to a missed sharing step, not to code.

A 429 means slow down, not try harder. Read the Retry After header when present, then wait with exponential backoff and jitter before retrying. A common pattern waits 2 seconds, then 4, then 8, then 16, with a small random addition to avoid synchronized retries. Cap retries at 4 or 5 and move the URL to a delayed queue after that. Hammering the endpoint during a limit only extends the block and burns log space.

In practice, create a short runbook for what 429 means on this api and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

Daily quota versus per minute throttle#

This section covers daily quota versus per minute throttle in the context of indexing api 429 error. Two limiters work together. Daily quota caps total publishes per 24 hour window and resets around midnight Pacific, while per minute throttling smooths bursts and clears after a short pause. If 429 persists all day with zero successes, daily allowance is likely spent. If errors cluster in minutes around deploys then clear, you hit the per minute limiter. Check Cloud Console graphs by method to tell which one you met, then choose deferral or pacing accordingly. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Crawl budget is often misunderstood. For small sites it rarely limits indexing, but for catalogs with 50,000 to 500,000 URLs it shapes what gets visited each day. Facets, session parameters, internal search results and duplicate variants can trap crawlers in low value loops. Use robots rules to block filtered views, use canonical tags to consolidate variants, and link best sellers from the home page and category hubs. Fewer dead ends means faster visits to new and updated pages.

• Step 1: Confirm Cloud project ID, service account email, and active key ID in IAM.
• Step 2: Confirm the Indexing API is enabled in the API library for that project.
• Step 3: Confirm Search Console ownership on the exact property including scheme.
• Step 4: Send one metadata read, then one publish for a test URL, and inspect both responses.
• Step 5: Enable alerts at 60 and 85 percent of quota before resuming bulk work.

The Google Indexing API only documents JobPosting and BroadcastEvent pages, which covers job listings and livestream video. Many site owners still test it for product or article URLs, but that use is off label and results vary. Google may process the hint, ignore it, or throttle it. State this plainly to stakeholders. Use the API for eligible content first, and rely on sitemaps, internal links and IndexNow for broad coverage on other page types.

A 403 usually points to permissions or scope. Confirm the service account email has Owner access in Search Console, confirm the OAuth scope includes the indexing scope, and confirm the JSON key file matches the active key in Cloud Console. Check clock skew on the server, since JWT auth fails when time drifts by more than a few minutes. Rotate keys on a schedule, store them in a secret manager, and never paste private keys into chat tools or shared docs.

In practice, create a short runbook for daily quota versus per minute throttle and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

For background on a related setup, see quota limits explained which explains how publishers structure notifications and sitemaps for time sensitive pages.

Reading the Retry After header#

This section covers reading the retry after header in the context of indexing api 429 error. When present, Retry After gives seconds or an HTTP date to wait before the next attempt. Honor it exactly, then add a small buffer rather than retrying a second early. If the header is absent, start with a conservative wait such as 30 to 60 seconds for per minute blocks and longer for daily exhaustion. Log the header value with each 429 so tuning is based on evidence. Clients that respect this signal recover faster and trigger fewer repeat blocks. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Respect the retry-after header exactly and pair it with indexing api backoff so recovery uses evidence. Honor seconds or dates plus a small buffer, then resume at half pace with full logging. This api retry logic avoids early retries that extend blocks and keeps tuning tied to logged header values.

The Google Indexing API only documents JobPosting and BroadcastEvent pages, which covers job listings and livestream video. Many site owners still test it for product or article URLs, but that use is off label and results vary. Google may process the hint, ignore it, or throttle it. State this plainly to stakeholders. Use the API for eligible content first, and rely on sitemaps, internal links and IndexNow for broad coverage on other page types.

Google does not support IndexNow, so plan for two ecosystems. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for eligible types. A practical setup sends product updates to both paths at publish time. One worker prepares the URL list, then one branch pings IndexNow endpoints and another branch queues Google notifications within quota. Coverage improves without double counting.

JWT failures look cryptic but follow a pattern. Invalid signature often means the wrong key file or a corrupted newline in the private key. Invalid grant often means the service account is disabled or the project never enabled the API. Failed to parse often means the token was truncated in logs or copied with extra spaces. Keep token lifetimes short, request a fresh access token per batch, and log the key ID without logging the secret. Small hygiene steps remove most auth noise.

In practice, create a short runbook for reading the retry after header and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

For official details, see developers.google.com docs which documents the expected fields and behavior.

Exponential backoff with jitter for indexing api 429 error#

This section covers exponential backoff with jitter in the context of indexing api 429 error. Exponential backoff spaces retries as 2, 4, 8, then 16 seconds, with a small random jitter to avoid synchronized waves from parallel workers. Cap retries at four or five attempts, then move the URL to a delayed queue instead of looping forever. Jitter matters because ten workers retrying on the same second recreate the burst that caused the block. Keep backoff in a shared helper so every language path and every cron job uses the same timing rules. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Quotas shape every automation decision. Many projects start with about 200 publish requests per day for URL notifications, plus per minute limits that trigger 429 when bursts arrive. Track usage in Cloud Console under APIs and Services, set alerts at 60 percent and 85 percent, and log each publish with timestamp, URL, response code and notification type. When you know your burn rate by hour, you can pace jobs, defer low priority URLs and avoid midnight surprises.

Quotas shape every automation decision. Many projects start with about 200 publish requests per day for URL notifications, plus per minute limits that trigger 429 when bursts arrive. Track usage in Cloud Console under APIs and Services, set alerts at 60 percent and 85 percent, and log each publish with timestamp, URL, response code and notification type. When you know your burn rate by hour, you can pace jobs, defer low priority URLs and avoid midnight surprises.

Logging turns guesses into fixes. For each submission store the URL, notification type, HTTP status, response body snippet, latency and a correlation ID. Keep success and error logs separate so you can scan error rates by hour. Export daily counts to a sheet or dashboard that shows submits, 200 responses, 403 responses, 429 responses and remaining quota. When stakeholders ask why a product is not visible, you can point to exact evidence instead of general theories.

In practice, create a short runbook for exponential backoff with jitter and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

To compare paths side by side, read rate limits and fixes before you commit quota to one route.

Queue design that prevents bursts#

This section covers queue design that prevents bursts in the context of indexing api 429 error. A durable queue absorbs publish spikes from CMS saves, imports, and deploys. Use a table or stream with URL, notification type, priority, attempts, next retry time, and last status. A single worker pulls at a fixed rate such as one request every few seconds, updates status per response, and requeues 429 with a future timestamp. Separate high priority updates for eligible pages from low priority refreshes so limits serve important URLs first when capacity is tight. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Durable api queue design with a single paced worker makes it simple to handle 429 without manual scrambles. Pull at a fixed rate, prioritize eligible updates, and requeue throttled URLs with future timestamps. This indexing api wait pattern absorbs CMS spikes while keeping volume under daily and per minute caps.

A 403 usually points to permissions or scope. Confirm the service account email has Owner access in Search Console, confirm the OAuth scope includes the indexing scope, and confirm the JSON key file matches the active key in Cloud Console. Check clock skew on the server, since JWT auth fails when time drifts by more than a few minutes. Rotate keys on a schedule, store them in a secret manager, and never paste private keys into chat tools or shared docs.

• Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
• Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
• Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
• Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
• Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

A 429 means slow down, not try harder. Read the Retry After header when present, then wait with exponential backoff and jitter before retrying. A common pattern waits 2 seconds, then 4, then 8, then 16, with a small random addition to avoid synchronized retries. Cap retries at 4 or 5 and move the URL to a delayed queue after that. Hammering the endpoint during a limit only extends the block and burns log space.

Internal linking does more for indexing than most teams expect. New URLs that sit four clicks from the home page may wait days for a visit, while URLs linked from a popular category or a recent posts block get visited quickly. Add new products to relevant category pages, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple, stable links help both Google and IndexNow driven crawlers find changes fast.

In practice, create a short runbook for queue design that prevents bursts and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

import time, random
def backoff(attempt, retry_after=None):
    if retry_after:
        time.sleep(float(retry_after) + 1)
        return
    base = min(2 ** attempt, 16)
    time.sleep(base + random.uniform(0, 1))
# call backoff(n) on 429, cap at 5 attempts then delay queue row

Pacing bulk jobs over hours#

This section covers pacing bulk jobs over hours in the context of indexing api 429 error. Bulk migrations and catalog imports can exceed daily allowance in minutes if sent in a tight loop. Split the list into hourly batches sized to your quota headroom, deduplicate first, and drop 404 and noindex URLs before queueing. Run the queue during off peak hours, monitor success rate per hour, and pause automatically when 429 exceeds a threshold. Spreading ten thousand URLs over days with logging beats hammering the endpoint and stalling for a full reset cycle. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Logging turns guesses into fixes. For each submission store the URL, notification type, HTTP status, response body snippet, latency and a correlation ID. Keep success and error logs separate so you can scan error rates by hour. Export daily counts to a sheet or dashboard that shows submits, 200 responses, 403 responses, 429 responses and remaining quota. When stakeholders ask why a product is not visible, you can point to exact evidence instead of general theories.

A 403 usually points to permissions or scope. Confirm the service account email has Owner access in Search Console, confirm the OAuth scope includes the indexing scope, and confirm the JSON key file matches the active key in Cloud Console. Check clock skew on the server, since JWT auth fails when time drifts by more than a few minutes. Rotate keys on a schedule, store them in a secret manager, and never paste private keys into chat tools or shared docs.

Canonical tags decide which URL keeps the indexing credit. If variants with color, size or tracking parameters lack a canonical, Google may pick a different URL or delay indexing while it compares duplicates. Point each variant to the preferred canonical, keep the canonical self referencing on the main URL, and make sure sitemaps list only canonicals. For translated or regional pages, add hreflang and keep each locale self consistent. Clean signals shorten the decision time.

In practice, create a short runbook for pacing bulk jobs over hours and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

Logging and alerting for throttle events#

This section covers logging and alerting for throttle events in the context of indexing api 429 error. Throttle visibility starts with per request logs: time, URL, notification type, status, Retry After, and worker ID. Aggregate to hourly counts of 200, 403, 404, 429, and 500 so patterns appear without reading raw lines. Alert at rising 429 rate and at 60 and 85 percent of daily quota, not only at full exhaustion. Dashboards that show burn rate by hour let owners defer low priority batches before the ceiling is hit. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Canonical tags decide which URL keeps the indexing credit. If variants with color, size or tracking parameters lack a canonical, Google may pick a different URL or delay indexing while it compares duplicates. Point each variant to the preferred canonical, keep the canonical self referencing on the main URL, and make sure sitemaps list only canonicals. For translated or regional pages, add hreflang and keep each locale self consistent. Clean signals shorten the decision time.

JWT failures look cryptic but follow a pattern. Invalid signature often means the wrong key file or a corrupted newline in the private key. Invalid grant often means the service account is disabled or the project never enabled the API. Failed to parse often means the token was truncated in logs or copied with extra spaces. Keep token lifetimes short, request a fresh access token per batch, and log the key ID without logging the secret. Small hygiene steps remove most auth noise.

Robots directives and meta tags can silently block indexing. A stray noindex in a template, an X-Robots-Tag header from a staging config, or a disallow in robots that covers new paths will keep pages out even after successful submission. Audit headers with a fetch tool, render pages as Googlebot, and check the coverage report for Excluded by noindex or Blocked by robots. Fix the template once rather than patching URLs one by one.

In practice, create a short runbook for logging and alerting for throttle events and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

When errors persist, review quota exceeded recovery playbook to isolate auth, access and pacing causes with logs.

Recovery after a long block#

This section covers recovery after a long block in the context of indexing api 429 error. When daily quota is spent, stop new publishes, preserve order in the queue, and switch the site to sitemap and internal link coverage until reset. Revalidate the backlog after the pause: remove URLs that changed state, became noindex, or gained canonical variants. Resume at half pace with full logging, then ramp only after an hour of clean 200 responses. Document the incident with cause, counts, and pacing change so the next launch does not repeat the burst. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

After repeated 429 too many requests pauses, triage rate limit errors by checking daily versus per minute graphs before resuming. Slow throttling submissions with hourly batches, deduplication and deferral of low value URLs. This 429 fix google style routine documents cause, counts and pacing change so launches do not repeat the burst.

Thin or duplicated content slows indexing because Google prioritizes pages likely to satisfy searchers. Short product descriptions copied from suppliers, empty category pages and near duplicate articles often sit in Discovered or Crawled without indexing. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing.

• Step 1: Confirm Cloud project ID, service account email, and active key ID in IAM.
• Step 2: Confirm the Indexing API is enabled in the API library for that project.
• Step 3: Confirm Search Console ownership on the exact property including scheme.
• Step 4: Send one metadata read, then one publish for a test URL, and inspect both responses.
• Step 5: Enable alerts at 60 and 85 percent of quota before resuming bulk work.

Logging turns guesses into fixes. For each submission store the URL, notification type, HTTP status, response body snippet, latency and a correlation ID. Keep success and error logs separate so you can scan error rates by hour. Export daily counts to a sheet or dashboard that shows submits, 200 responses, 403 responses, 429 responses and remaining quota. When stakeholders ask why a product is not visible, you can point to exact evidence instead of general theories.

Thin or duplicated content slows indexing because Google prioritizes pages likely to satisfy searchers. Short product descriptions copied from suppliers, empty category pages and near duplicate articles often sit in Discovered or Crawled without indexing. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing.

In practice, create a short runbook for recovery after a long block and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

import time, random
def backoff(attempt, retry_after=None):
    if retry_after:
        time.sleep(float(retry_after) + 1)
        return
    base = min(2 ** attempt, 16)
    time.sleep(base + random.uniform(0, 1))
# call backoff(n) on 429, cap at 5 attempts then delay queue row

When to defer low priority URLs#

This section covers when to defer low priority urls in the context of indexing api 429 error. Not every changed URL deserves scarce quota. Prioritize new eligible JobPosting and BroadcastEvent pages, price and availability changes, and corrected canonicals. Defer evergreen refreshes, paginated archives, and faceted variants to sitemap discovery. Score each URL by age, change size, and business value, then sort the queue by that score. Explicit deferral rules keep the most valuable pages moving while total volume stays under the daily cap. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Sitemaps remain the backbone of discovery. A clean product or article sitemap lists only canonical, indexable URLs that return 200 and load quickly. Split large catalogs into chunks of 10,000 to 40,000 URLs, compress with gzip, and reference each chunk from a sitemap index. Update the lastmod field only when content truly changes. Submit the index in Search Console and keep it reachable. A tidy sitemap reduces wasted fetches and leaves room for priority pages.

Internal linking does more for indexing than most teams expect. New URLs that sit four clicks from the home page may wait days for a visit, while URLs linked from a popular category or a recent posts block get visited quickly. Add new products to relevant category pages, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple, stable links help both Google and IndexNow driven crawlers find changes fast.

Google discovers most pages through crawl, not through a single submission. A submission is a hint that asks for a fresh look, but ranking and storage still depend on quality, uniqueness and site trust. That is why steady technical hygiene matters more than any one push. Keep response times low, avoid redirect chains, and return clear status codes. When the crawler can fetch quickly and without loops, each hint carries more weight and uses less of your daily allowance.

In practice, create a short runbook for when to defer low priority urls and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

For HTTP semantics, see status code reference which defines how clients should handle each response.

Code for resilient submission workers#

This section covers code for resilient submission workers in the context of indexing api 429 error. A resilient worker combines rate limiting, backoff, and idempotency. It leases one row at a time, sends the publish call, handles 200 as done, 429 with delayed retry, 403 with a pause for review, and 404 as done with a note. It records every outcome and never deletes a row without a terminal status. Timeouts are short, retries are bounded, and concurrency is fixed to one or two slots. This shape survives deploys and restarts without duplicating sends or losing order. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Search Console verification is the gate for any Google submission. The service account that calls the API must be added as an Owner on the exact property, including the correct scheme and subdomain. Domain properties and URL prefix properties behave differently, so match the property you verify with the URLs you submit. If you see permission denied or 403, check sharing settings first, then OAuth scope, then key expiry. Most auth failures trace to a missed sharing step, not to code.

Canonical tags decide which URL keeps the indexing credit. If variants with color, size or tracking parameters lack a canonical, Google may pick a different URL or delay indexing while it compares duplicates. Point each variant to the preferred canonical, keep the canonical self referencing on the main URL, and make sure sitemaps list only canonicals. For translated or regional pages, add hreflang and keep each locale self consistent. Clean signals shorten the decision time.

Sitemaps remain the backbone of discovery. A clean product or article sitemap lists only canonical, indexable URLs that return 200 and load quickly. Split large catalogs into chunks of 10,000 to 40,000 URLs, compress with gzip, and reference each chunk from a sitemap index. Update the lastmod field only when content truly changes. Submit the index in Search Console and keep it reachable. A tidy sitemap reduces wasted fetches and leaves room for priority pages.

In practice, create a short runbook for code for resilient submission workers and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

A checklist to stay under the ceiling#

This section covers a checklist to stay under the ceiling in the context of indexing api 429 error. Keep a one page routine: track daily usage, pace workers to a fixed interval, deduplicate before queueing, prioritize eligible pages, honor Retry After, alert at 60 and 85 percent, and review logs weekly. Assign one owner for quota and one for queue health. Test pacing on staging with a small batch before each large import. Short, repeated checks prevent most 429 incidents before they start and make recovery uneventful when limits are met. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with Search Console, server logs and a small script. The goal is steady progress you can measure in coverage reports, not a one time spike. Keep notes on what you change and when, so you can link indexing movement to specific fixes.

Google does not support IndexNow, so plan for two ecosystems. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for eligible types. A practical setup sends product updates to both paths at publish time. One worker prepares the URL list, then one branch pings IndexNow endpoints and another branch queues Google notifications within quota. Coverage improves without double counting.

• Step 1: Export the full URL list from your CMS or commerce platform with last change dates.
• Step 2: Join with Search Console coverage to flag Discovered, Crawled, Excluded and Indexed states.
• Step 3: Remove duplicates, 404s, noindex pages and non canonical variants from the submit set.
• Step 4: Sort by priority such as new, price change, availability change, then evergreen refresh.
• Step 5: Queue at a paced rate, log every response, and pause on repeated 429 or 403.

Robots directives and meta tags can silently block indexing. A stray noindex in a template, an X-Robots-Tag header from a staging config, or a disallow in robots that covers new paths will keep pages out even after successful submission. Audit headers with a fetch tool, render pages as Googlebot, and check the coverage report for Excluded by noindex or Blocked by robots. Fix the template once rather than patching URLs one by one.

Crawl budget is often misunderstood. For small sites it rarely limits indexing, but for catalogs with 50,000 to 500,000 URLs it shapes what gets visited each day. Facets, session parameters, internal search results and duplicate variants can trap crawlers in low value loops. Use robots rules to block filtered views, use canonical tags to consolidate variants, and link best sellers from the home page and category hubs. Fewer dead ends means faster visits to new and updated pages.

In practice, create a short runbook for a checklist to stay under the ceiling and review it after each deploy. List who owns keys, who owns Search Console, where logs live, and what alert fires first. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady, documented pacing restores submissions faster than rushing to catch up in one burst.

FAQ#

What causes 429 too many requests on the Indexing API?#

An 429 too many requests response means you sent faster than daily or per minute allowance permits, not that URLs are invalid. Check Cloud Console graphs by method to tell daily exhaustion from burst throttling. Preserve backlog in the queue, pause new publishes and resume slower with backoff. These rate limit errors often cluster around deploys and imports. Log status, Retry After and worker ID per request so tuning uses evidence. This indexing api wait discipline restores steady 200 responses faster than immediate retries.

How should indexing api backoff honor the retry-after header?#

Good indexing api backoff honors the retry-after header exactly plus a small buffer instead of retrying a second early. When the header is present, wait the stated seconds or until the given date, then resume at half pace. Without it, wait 30 to 60 seconds for burst blocks and until daily reset for exhausted quota. Log each header value with its 429 to tune pacing. This api retry logic respects the limiter, triggers fewer repeat blocks and keeps handle 429 playbooks simple for on call staff.

What is exponential backoff with jitter for throttling submissions?#

Exponential backoff spaces retries as 2, 4, 8 then 16 seconds with random jitter to avoid synchronized waves when throttling submissions across parallel workers. Cap attempts at four or five, then move the URL to a delayed queue instead of looping. Jitter matters because ten workers retrying on the same second recreate the burst. Keep this api retry logic in a shared helper so every cron job uses the same timing. This indexing api backoff pattern lowers 429 fix google searches and stabilizes hourly throughput.

Can api queue design prevent 429?#

Yes. A durable api queue design absorbs CMS spikes, imports and deploys that would otherwise trigger limits. Use a table with URL, notification type, priority, attempts, next retry time and last status. A single worker pulls at a fixed rate, updates status per response and requeues 429 with future timestamps. Prioritize eligible new pages over evergreen refreshes when capacity is tight. This indexing api wait strategy keeps volume under caps and makes handle 429 actions automatic rather than manual scrambles.

Should I retry 429 immediately or use indexing api wait?#

No. Immediate retries extend the block, so use indexing api wait discipline instead. Honor Retry After, lower the rate and move repeated failures to a delayed queue. Resume only after clean responses return for a full hour. To handle 429 safely, track 429 fix google style runbooks with cause, counts and pacing change for each incident. Keep concurrency at one or two slots, add timeouts and never delete rows without terminal status. Calm pacing beats rushing to catch up in one burst.

When should low priority URLs wait during rate limit errors?#

When rate limit errors rise, serve new eligible pages and important updates first and defer evergreen refreshes, archives and faceted variants to sitemap discovery. Score each URL by age, change size and business value, then sort the queue by that score. This throttling submissions triage keeps valuable pages moving under daily caps. Review burn rate hourly, alert at 60 and 85 percent of quota and record deferral rules in a one page routine so launches do not repeat the same burst.

Sources#

• https://developers.google.com/search/apis/indexing-api/v3/errors
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
• https://developers.google.com/search/apis/indexing-api/v3/prereqs

Further reading#

• Indexing API errors for 403 and JWT
• bulk submission safe limits
• status codes field guide