Indexer by DependsiT

How to Build a Self-Hosted IndexNow Submitter

Self hosted IndexNow submitter code blocks and queue flow on dark

This guide is for site owners, developers and SEO leads who work with self hosted indexnow and need a reliable routine without guesswork. Many teams hit the same wall: submissions return mixed codes, logs are thin, and regional or automated coverage moves slowly while stakeholders ask for dates. The facts matter here. IndexNow is an open protocol co developed by Microsoft Bing and Yandex, Google does not support IndexNow, and one request can carry up to 10000 URLs with a key file at the site root for verification. You will learn exact checks, safe pacing, logging that proves what happened, and recovery steps that work in production. Follow the sections in order, test with a handful of URLs first, then scale once responses stay clean.

Intro scope: this article focuses on practical IndexNow operation for production sites. It assumes a live https host, access to server logs, and the ability to host a plain text file at the site root. It does not promise rankings or instant inclusion, because engines still decide crawl and index eligibility after a successful ping.

Key takeaways

  • What a self hosted submitter does and does not do starts with key file health and host pure batches, not with faster retries.
  • Choosing PHP Python Node or plain cURL works best with one test URL and full logs before bulk batches.
  • IndexNow is co developed by Bing and Yandex, Google does not support it, and one request can carry up to 10000 URLs.
  • Deduplicate, filter to changed canonical URLs, then queue at a fixed pace with backoff on 429.

Self hosted IndexNow submitter code blocks and queue flow on dark <!-- IMAGE-PROMPT cover: 1200x630, DependsIt brand, deep charcoal #121212 background with vibrant mint #22E3B0 accent glow, thin node-network line art, Clash Display style bold heading space on left, General Sans clean labels, subject: self hosted indexnow cover illustration, flat vector, high contrast, accessible, no photorealistic faces, no text smaller than 24px, no em dash in rendered text, export PNG then cwebp -q 82 to WEBP -->

What a self hosted indexnow submitter does and does not do

This section covers what a self hosted submitter does and does not do in the context of self hosted indexnow. A self hosted submitter sends IndexNow JSON with host plus key plus keyLocation plus urlList to a participating endpoint and logs the response. It does not index pages, boost rankings or reach Google, because Google does not support IndexNow. Its value is control: keys stay on your server, logs stay in your account, and pacing follows your rules. Keep the tool small, readable and boring so any teammate can operate it during launches. Many owners start from a minimal indexnow submitter script when they decide to host your own indexer, then evolve it into a small custom indexing tool with queue and logs. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

IndexNow is a simple notification protocol co developed by Microsoft Bing and Yandex. When you create, update or delete a page, your site sends an HTTP request with the list of affected URLs plus a key that proves ownership. Participating engines can then prioritize those URLs for recrawling. The protocol does not upload content, does not guarantee indexing, and does not change quality evaluation. It shortens discovery time so good pages can be evaluated sooner with less waiting.

ItemWhat to recordWhere to check
RequestURL plus hostWorker log row
KeyFingerprint plus locationRoot file GET
ResponseStatus plus snippetEngine reply body
Follow upNext retry timeQueue next run field

Google does not support IndexNow, so plan for two ecosystems from the start. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for JobPosting and BroadcastEvent pages only. A practical setup prepares one URL list at publish time, then branches to IndexNow batches plus Google sitemap coverage. Coverage improves without double counting or false expectations.

Server logs prove whether IndexNow prompts faster fetches. Record submission time, endpoint, host, URL count and response code for every batch, then watch for engine fetches of the key file and notified URLs. Compare submitted URLs against an unsubmitted control group from the same template to measure delay honestly. If fetch quickens but index state does not change, the constraint is content or eligibility rather than discovery. Evidence beats assumptions in every review.

In practice, create a short runbook for what a self hosted submitter does and does not do and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

Choosing PHP Python Node or plain cURL

This section covers choosing php python node or plain curl in the context of self hosted indexnow. Pick the stack your team already runs. PHP fits shared hosting and WordPress servers, Python fits small workers and cron jobs, Node fits JavaScript pipelines, and cURL fits one off tests and shell hooks. All four can POST JSON, read response codes and write logs. Avoid adding frameworks for a tool that needs one HTTP call plus a queue. Fewer dependencies means fewer midnight breakages. A simple indexnow php script suits shared hosting, a compact indexnow python worker suits cron jobs, and an indexnow curl script suits shell tests before automation. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Ownership in IndexNow is proven by a key text file hosted at the site root. The file name is your key plus .txt and its content is the key string itself. Engines fetch that URL during verification and compare it with the key in your submission. If the file is missing, returns 404, wraps content in HTML, or blocks anonymous fetch, verification fails with client errors. Keep the file plain text, publicly reachable, and stable across deploys and redesigns.

  • Step 1: Confirm host, key, keyLocation and endpoint in server config.
  • Step 2: Validate JSON shape locally against the official docs before sending.
  • Step 3: Deduplicate URLs and split by host so each request stays host pure.
  • Step 4: Send at a fixed pace, handle 429 with backoff and 403 with pause for review.
  • Step 5: Review fetch logs daily and record submit to fetch delay per URL group.

One IndexNow POST can carry up to 10000 URLs in urlList, but polite batches of a few hundred with pauses work better for steady trust. Deduplicate before sending, keep requests host pure with absolute https URLs on one host, and split larger backlogs across requests with delays. Reject invalid entries at queue entry with a logged skip reason so one bad URL never fails a batch of good ones. Honest change only signals beat full sitemap dumps sent daily.

Crawl health still decides whether a notified URL qualifies for the index. Keep response times low, avoid redirect chains, return clear status codes, and allow anonymous GET for important resources. Use canonical tags to consolidate variants, hreflang for translated pages, and robots rules that block only low value paths such as facets and internal search. When the crawler can fetch quickly without loops, each IndexNow hint carries more weight.

In practice, create a short runbook for choosing php python node or plain curl and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

For background on the protocol, see IndexNow complete guide which explains endpoints, keys and fanout across participating engines.

Diagram showing self hosted indexnow flow with key file, queue and engine fanout <!-- IMAGE-PROMPT diagram-01: 1600px max, DependsIt brand mint #22E3B0 on charcoal #121212, node-network line art, Clash Display style headings feel with General Sans clean labels, subject: self hosted indexnow diagram with key verification and crawl nodes, flat vector, accessible, no em dash in rendered text -->

Key handling and config layout

This section covers key handling and config layout in the context of self hosted indexnow. Store host, key, keyLocation and endpoint in environment or a secrets file outside the web root, never in Git or front end JavaScript. Keep one config per host, with a fingerprint of the key in logs rather than the raw string where possible. Document where the key file lives at the site root and when it was last verified. Clean config separation prevents staging keys from pinging production URLs. When you add indexnow automation code later, keep the same config shape so an indexnow webhook can queue URLs and each indexnow endpoint call reuses host plus key plus keyLocation safely. Document how to build indexing script changes in one place. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

One IndexNow POST can carry up to 10000 URLs in urlList, but polite batches of a few hundred with pauses work better for steady trust. Deduplicate before sending, keep requests host pure with absolute https URLs on one host, and split larger backlogs across requests with delays. Reject invalid entries at queue entry with a logged skip reason so one bad URL never fails a batch of good ones. Honest change only signals beat full sitemap dumps sent daily.

Response codes tell you exactly what to do next. Codes 200 and 202 mean accepted so you mark the batch done. Code 400 means malformed JSON so you fix shape locally. Code 403 means key or ownership failed so you verify the key file and host. Code 422 means invalid URLs so you clean the list. Code 429 means slow down so you back off with delay. Log code plus URL count plus response snippet for every send to make weekly triage fast.

Internal linking helps IndexNow driven crawlers find changes fast after the ping. New URLs that sit four clicks from the home page may wait longer for a visit, while URLs linked from popular categories or recent posts blocks get visited quickly. Add new pages to relevant hubs, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple stable links support both human visitors and engine crawlers.

In practice, create a short runbook for key handling and config layout and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

For official details, see IndexNow documentation which defines fields, key handling and batch rules.

Single URL submit function that you can trust

This section covers single url submit function that you can trust in the context of self hosted indexnow. Start with one function that submits a single absolute URL, checks that host matches config, validates the key file URL, sends the request with a short timeout, and returns status plus body. Test it with a harmless docs URL and confirm 200 or 202 plus a later bot fetch in logs. A correct single send proves auth, shape and network before batch logic adds complexity. Keep this function as the health check for deploys. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Bing Webmaster Tools gives the clearest reporting for IndexNow among Western engines, with submission views and crawl data to pair with your own logs. Yandex Webmaster provides its own verification and diagnostics for its index. Naver and Seznam publish less detailed quota text, so operate conservatively with small batches and strict change only filtering. Always re check the current participants page on the official site during planning rather than trusting a stale screenshot.

ItemWhat to recordWhere to check
RequestURL plus hostWorker log row
KeyFingerprint plus locationRoot file GET
ResponseStatus plus snippetEngine reply body
Follow upNext retry timeQueue next run field

Bing Webmaster Tools gives the clearest reporting for IndexNow among Western engines, with submission views and crawl data to pair with your own logs. Yandex Webmaster provides its own verification and diagnostics for its index. Naver and Seznam publish less detailed quota text, so operate conservatively with small batches and strict change only filtering. Always re check the current participants page on the official site during planning rather than trusting a stale screenshot.

Robots directives and meta tags can silently block indexing even after successful submission. A stray noindex in a template, an X Robots Tag header from a staging config, or a disallow that covers new paths will keep pages out. Audit headers with a fetch tool, render pages as a crawler sees them, and check webmaster coverage for excluded reasons. Fix the template once rather than patching URLs one by one. Clean permissions make each accepted ping useful.

In practice, create a short runbook for single url submit function that you can trust and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

To compare coverage and limits, read generate and host your IndexNow key before you commit time to one route.

Batch submit up to 10000 URLs without spam

This section covers batch submit up to 10000 urls without spam in the context of self hosted indexnow. One IndexNow request can carry up to 10000 URLs, but polite batches of a few hundred with pauses work better for steady trust. Deduplicate, keep one host per request, drop relative paths and non https entries, and split larger backlogs across requests with delays. Reject bad entries at queue entry with a logged skip reason so one bad URL never fails a batch of good ones. Honest batches beat full sitemap blasts. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Crawl health still decides whether a notified URL qualifies for the index. Keep response times low, avoid redirect chains, return clear status codes, and allow anonymous GET for important resources. Use canonical tags to consolidate variants, hreflang for translated pages, and robots rules that block only low value paths such as facets and internal search. When the crawler can fetch quickly without loops, each IndexNow hint carries more weight.

  • Step 1: Export changed URLs from CMS or build manifest with last change dates.
  • Step 2: Keep only canonical 200 URLs on one host, drop drafts, 404s and noindex pages.
  • Step 3: Confirm key file GET returns 200 with exact body from outside the network.
  • Step 4: Send a single test URL, confirm success code, check logs for engine fetch.
  • Step 5: Queue remaining URLs in small batches with pauses, log every response.

Server logs prove whether IndexNow prompts faster fetches. Record submission time, endpoint, host, URL count and response code for every batch, then watch for engine fetches of the key file and notified URLs. Compare submitted URLs against an unsubmitted control group from the same template to measure delay honestly. If fetch quickens but index state does not change, the constraint is content or eligibility rather than discovery. Evidence beats assumptions in every review.

Thin or duplicated content slows indexing because engines prioritize pages likely to satisfy searchers. Short descriptions copied from suppliers, empty category pages and near duplicate articles often wait longer for inclusion. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing after discovery.

In practice, create a short runbook for batch submit up to 10000 urls without spam and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

import json, urllib.request
HOST='example.com'
KEY='YOUR_KEY'
KEYLOC='https://example.com/YOUR_KEY.txt'
ENDPOINT='https://www.bing.com/indexnow'
body={'host':HOST,'key':KEY,'keyLocation':KEYLOC,'urlList':['https://example.com/docs/1','https://example.com/docs/2']}
req=urllib.request.Request(ENDPOINT, data=json.dumps(body).encode(), headers={'Content-Type':'application/json'})
print(urllib.request.urlopen(req, timeout=20).status)

Response handling for 200 202 400 403 422 429

This section covers response handling for 200 202 400 403 422 429 in the context of self hosted indexnow. Map each code to one action: 200 and 202 mean accepted so mark done, 400 means malformed JSON so fix shape locally, 403 means key or permission issue so verify the key file, 422 means invalid URLs so clean the list, 429 means slow down so back off with delay. Log code plus URL count plus response snippet for every send. Consistent mapping turns confusing failures into short fixes. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Robots directives and meta tags can silently block indexing even after successful submission. A stray noindex in a template, an X Robots Tag header from a staging config, or a disallow that covers new paths will keep pages out. Audit headers with a fetch tool, render pages as a crawler sees them, and check webmaster coverage for excluded reasons. Fix the template once rather than patching URLs one by one. Clean permissions make each accepted ping useful.

Crawl health still decides whether a notified URL qualifies for the index. Keep response times low, avoid redirect chains, return clear status codes, and allow anonymous GET for important resources. Use canonical tags to consolidate variants, hreflang for translated pages, and robots rules that block only low value paths such as facets and internal search. When the crawler can fetch quickly without loops, each IndexNow hint carries more weight.

Security for IndexNow stays simple because auth uses a key file rather than JWTs, but hygiene still matters. Keep the submitter config private on the server, never embed keys in browser JavaScript, and store host plus key plus endpoint outside the web root or in platform secrets. Serve the key text file at the root as plain text for engines while keeping submitter secrets separate. Rotate with overlap by hosting old and new files briefly during switches.

In practice, create a short runbook for response handling for 200 202 400 403 422 429 and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

Workflow showing self hosted indexnow handling with paced queue and logging <!-- IMAGE-PROMPT workflow-02: 1600px max, DependsIt brand mint #22E3B0 on charcoal #121212 or white, node-network line art, Clash Display style headings feel with General Sans clean labels, subject: self hosted indexnow workflow with retry and audit steps, flat vector, accessible, no em dash in rendered text -->

Queue logging and retry design that survives restarts

This section covers queue logging and retry design that survives restarts in the context of self hosted indexnow. A tiny queue table with URL, host, attempts, next retry time and last status absorbs CMS spikes and deploy restarts. One worker pulls due rows at a fixed pace, sends through the single submit function, and updates status per response. Retry 429 and 500 with backoff, pause on 403 for review, and mark 400 and 422 as needs fix rather than looping forever. Durable rows plus fixed pacing keep submissions calm. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Security for IndexNow stays simple because auth uses a key file rather than JWTs, but hygiene still matters. Keep the submitter config private on the server, never embed keys in browser JavaScript, and store host plus key plus endpoint outside the web root or in platform secrets. Serve the key text file at the root as plain text for engines while keeping submitter secrets separate. Rotate with overlap by hosting old and new files briefly during switches.

ItemWhat to recordWhere to check
RequestURL plus hostWorker log row
KeyFingerprint plus locationRoot file GET
ResponseStatus plus snippetEngine reply body
Follow upNext retry timeQueue next run field

Internal linking helps IndexNow driven crawlers find changes fast after the ping. New URLs that sit four clicks from the home page may wait longer for a visit, while URLs linked from popular categories or recent posts blocks get visited quickly. Add new pages to relevant hubs, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple stable links support both human visitors and engine crawlers.

Automation works best with a queue between publish events and engine endpoints. Hooks in the CMS or deploy pipeline insert rows in milliseconds while a single worker sends at a fixed pace such as one batch per minute. Filter to changed canonical indexable URLs only, dropping drafts, previews, archives and staging domains. Fixed pacing plus retry timestamps absorbs import spikes, prevents limit storms, and keeps logs readable during launches.

In practice, create a short runbook for queue logging and retry design that survives restarts and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

When errors persist, review IndexNow bulk 10000 URL rule to isolate key, shape and pacing causes with logs.

Security file perms secrets and no key in frontend

This section covers security file perms secrets and no key in frontend in the context of self hosted indexnow. Run the submitter server side only, with config files readable by the app user alone and logs that redact raw keys. Never embed the key in browser JavaScript, mobile apps or public repos. Serve the key text file at the root as plain text for engines, but keep the submitter config separate and private. Rotate the key with overlap by hosting old and new files briefly, then switching config after verification. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Measurement should separate discovery speed from ranking movement. IndexNow speeds the first step from change to engine fetch, but ranking still depends on relevance, quality and competition. Track submit to fetch delay in logs, fetch to index state in webmaster tools, and index to visit in analytics as three separate intervals. When stakeholders ask about value, report each interval with dates rather than promising position gains that the protocol never claims.

  • Step 1: Confirm host, key, keyLocation and endpoint in server config.
  • Step 2: Validate JSON shape locally against the official docs before sending.
  • Step 3: Deduplicate URLs and split by host so each request stays host pure.
  • Step 4: Send at a fixed pace, handle 429 with backoff and 403 with pause for review.
  • Step 5: Review fetch logs daily and record submit to fetch delay per URL group.

Robots directives and meta tags can silently block indexing even after successful submission. A stray noindex in a template, an X Robots Tag header from a staging config, or a disallow that covers new paths will keep pages out. Audit headers with a fetch tool, render pages as a crawler sees them, and check webmaster coverage for excluded reasons. Fix the template once rather than patching URLs one by one. Clean permissions make each accepted ping useful.

Measurement should separate discovery speed from ranking movement. IndexNow speeds the first step from change to engine fetch, but ranking still depends on relevance, quality and competition. Track submit to fetch delay in logs, fetch to index state in webmaster tools, and index to visit in analytics as three separate intervals. When stakeholders ask about value, report each interval with dates rather than promising position gains that the protocol never claims.

In practice, create a short runbook for security file perms secrets and no key in frontend and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

import json, urllib.request
HOST='example.com'
KEY='YOUR_KEY'
KEYLOC='https://example.com/YOUR_KEY.txt'
ENDPOINT='https://www.bing.com/indexnow'
body={'host':HOST,'key':KEY,'keyLocation':KEYLOC,'urlList':['https://example.com/docs/1','https://example.com/docs/2']}
req=urllib.request.Request(ENDPOINT, data=json.dumps(body).encode(), headers={'Content-Type':'application/json'})
print(urllib.request.urlopen(req, timeout=20).status)

Testing locally before production traffic

This section covers testing locally before production traffic in the context of self hosted indexnow. Test with a local checklist: key file fetch returns exact body, single submit returns success, batch of ten returns success, invalid URL returns a clean skip, and logs show endpoint plus status plus count. Use a staging host with its own key before touching production. Record each test with date and result so future changes have a baseline. Five careful tests prevent most production incidents. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Sitemaps remain the backbone of discovery even with IndexNow in place. A clean XML sitemap lists only canonical indexable URLs that return 200 and load quickly. Split large catalogs into chunks, compress with gzip, and reference each chunk from a sitemap index. Update lastmod only when content truly changes. Submit the index in Bing Webmaster Tools and keep it reachable. A tidy sitemap reduces wasted fetches and leaves room for priority pages to be visited sooner.

Thin or duplicated content slows indexing because engines prioritize pages likely to satisfy searchers. Short descriptions copied from suppliers, empty category pages and near duplicate articles often wait longer for inclusion. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing after discovery.

IndexNow is a simple notification protocol co developed by Microsoft Bing and Yandex. When you create, update or delete a page, your site sends an HTTP request with the list of affected URLs plus a key that proves ownership. Participating engines can then prioritize those URLs for recrawling. The protocol does not upload content, does not guarantee indexing, and does not change quality evaluation. It shortens discovery time so good pages can be evaluated sooner with less waiting.

In practice, create a short runbook for testing locally before production traffic and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

For worker hosting guidance, see Cloudflare Workers docs which covers secrets, schedules and fetch POSTs.

Deploying on VPS Cloudflare Workers or shared hosting

This section covers deploying on vps cloudflare workers or shared hosting in the context of self hosted indexnow. On a VPS use cron plus a small Python or Node worker with secrets in the platform store. On Cloudflare Workers use scheduled handlers with secrets bindings and fetch for POSTs. On shared hosting use a PHP snippet hooked to publish events with a file based queue. In every case keep concurrency at one, timeouts short, and logs central. Match the deploy to skills your team already has. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Google does not support IndexNow, so plan for two ecosystems from the start. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for JobPosting and BroadcastEvent pages only. A practical setup prepares one URL list at publish time, then branches to IndexNow batches plus Google sitemap coverage. Coverage improves without double counting or false expectations.

ItemWhat to recordWhere to check
RequestURL plus hostWorker log row
KeyFingerprint plus locationRoot file GET
ResponseStatus plus snippetEngine reply body
Follow upNext retry timeQueue next run field

Security for IndexNow stays simple because auth uses a key file rather than JWTs, but hygiene still matters. Keep the submitter config private on the server, never embed keys in browser JavaScript, and store host plus key plus endpoint outside the web root or in platform secrets. Serve the key text file at the root as plain text for engines while keeping submitter secrets separate. Rotate with overlap by hosting old and new files briefly during switches.

Sitemaps remain the backbone of discovery even with IndexNow in place. A clean XML sitemap lists only canonical indexable URLs that return 200 and load quickly. Split large catalogs into chunks, compress with gzip, and reference each chunk from a sitemap index. Update lastmod only when content truly changes. Submit the index in Bing Webmaster Tools and keep it reachable. A tidy sitemap reduces wasted fetches and leaves room for priority pages to be visited sooner.

In practice, create a short runbook for deploying on vps cloudflare workers or shared hosting and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

Maintenance checklist for your submitter

This section covers maintenance checklist for your submitter in the context of self hosted indexnow. Review monthly: dependencies, key file reachability after CDN changes, log growth, error rates by code, and queue age. Re test with five URLs after each host migration or PHP upgrade. Keep a one page runbook with config path, endpoint, key fingerprint date, pacing and alert channel. Small, regular upkeep keeps a simple tool reliable for years. We keep the advice practical for owners without a large SEO team. Each step below uses plain checks you can run with webmaster tools, server logs and a small script. The goal is steady progress you can measure in fetch reports, not a one time spike. Keep notes on what you change and when, so you can link movement to specific fixes.

Response codes tell you exactly what to do next. Codes 200 and 202 mean accepted so you mark the batch done. Code 400 means malformed JSON so you fix shape locally. Code 403 means key or ownership failed so you verify the key file and host. Code 422 means invalid URLs so you clean the list. Code 429 means slow down so you back off with delay. Log code plus URL count plus response snippet for every send to make weekly triage fast.

  • Step 1: Export changed URLs from CMS or build manifest with last change dates.
  • Step 2: Keep only canonical 200 URLs on one host, drop drafts, 404s and noindex pages.
  • Step 3: Confirm key file GET returns 200 with exact body from outside the network.
  • Step 4: Send a single test URL, confirm success code, check logs for engine fetch.
  • Step 5: Queue remaining URLs in small batches with pauses, log every response.

Automation works best with a queue between publish events and engine endpoints. Hooks in the CMS or deploy pipeline insert rows in milliseconds while a single worker sends at a fixed pace such as one batch per minute. Filter to changed canonical indexable URLs only, dropping drafts, previews, archives and staging domains. Fixed pacing plus retry timestamps absorbs import spikes, prevents limit storms, and keeps logs readable during launches.

Ownership in IndexNow is proven by a key text file hosted at the site root. The file name is your key plus .txt and its content is the key string itself. Engines fetch that URL during verification and compare it with the key in your submission. If the file is missing, returns 404, wraps content in HTML, or blocks anonymous fetch, verification fails with client errors. Keep the file plain text, publicly reachable, and stable across deploys and redesigns.

In practice, create a short runbook for maintenance checklist for your submitter and review it after each deploy. List who owns the key file, where logs live, what alert fires first, and what pace the worker uses. Test with five URLs before scaling to hourly batches. Record every response code and timestamp so patterns appear without guesswork. If errors rise, pause automation, fix the root cause, then resume at half pace. Steady documented pacing restores submissions faster than rushing to catch up in one burst.

FAQ

What language should I use for a submitter?

Use what your team already operates. PHP suits shared hosting, Python suits cron workers, Node suits JS pipelines, and cURL suits tests. An indexnow php script is easiest on shared hosts, while a compact indexnow python worker gives cleaner logging for queues. All can POST JSON and log codes when written carefully. Test with five URLs before scaling to hourly batches for safety. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory.

Where do I store the IndexNow key?

Host the key text file at the site root for engines, and keep submitter config with the key string private on the server. Never commit it to Git or ship it in frontend code. A safe indexnow submitter script reads host plus key from environment, so you can host your own indexer without leaking secrets. Test with five URLs after each config change for safety. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory.

How many URLs per request is safe?

Up to 10000 is allowed, but a few hundred per request with pauses is calmer and easier to debug. Split larger backlogs and deduplicate before sending. Treat each batch as one indexnow endpoint call with logged status, and use a build indexing script checklist to keep host pure and skip bad rows. Review fetch logs daily and record submit to fetch delay per group. Keep a short log of what you checked and when, so the next review starts from evidence.

Does a self hosted tool reach Google?

No. Google does not support IndexNow. Use the tool for Bing, Yandex, Naver, Seznam and other partners, and keep Google work on sitemaps and Search Console paths. A custom indexing tool for IndexNow will never trigger Google fetches, so plan two tracks from the start to avoid false expectations. Document the split for stakeholders clearly and review coverage monthly with logs. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory and stays simple.

How do I test without risk?

Start with one harmless URL on a staging host, confirm success and key fetch, then try ten URLs. Check logs for endpoint, count and code before enabling automation. An indexnow curl script is ideal for this first probe, followed by an indexnow webhook test that queues one row end to end. Confirm bot fetch in logs before enabling publish hooks in production. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory and stays simple.

What maintenance does the tool need?

Monthly dependency checks, key file re verification after CDN changes, log rotation, error rate review and a five URL test after migrations. Keep a one page runbook with pacing and alerts for on call use. Teams that add indexnow automation code later should re test the queue after each deploy. Record every response code and timestamp so patterns appear without guesswork. Small regular upkeep keeps a simple tool reliable for years. Keep a short log of what you checked and when, so reviews start from evidence.

Sources

  • https://www.indexnow.org/documentation
  • https://developers.cloudflare.com/workers/
  • https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap

Further reading

Put this into practice. Indexer submits URLs to the Google Indexing API and IndexNow, audits coverage with Search Console, and shows exactly which pages are indexed. Start free or see how it works.