Reading the Crawl Stats Report in Search Console

This guide is for site owners, developers and SEOs who work with crawl stats report and need a clear routine without guesswork. Many teams see the same pattern: coverage reports stall, crawl stats swing, and stakeholders ask when priority pages will appear in results. The facts help here. Plan for efficient crawling, clean sitemaps, honest quality signals and steady measurement. You will learn exact checks, safe defaults, templates to copy and a review rhythm that fits a busy week. Follow the sections in order, test on a small sample first, then scale once responses and reports stay clean. The focus keyword crawl stats report appears where it helps mapping, never as filler.

Key takeaways

• Where to find the crawl stats report and who should use it sets the base: fast stable responses plus clean discovery signals for priority pages.
• How to read crawl purpose discovery refresh and recrawl matters most for large sites, where filters and variants consume visits.
• Track crawl stats, coverage and logs together for two week windows before judging a fix.
• Fix templates once, keep sitemaps accurate, and review monthly so gains hold.

• Where to find the crawl stats report and who should use it
• How to read total crawl requests and host load
• How to read crawl responses by type and what they signal
• How to read crawl purpose discovery refresh and recrawl
• How to spot server problems in crawl stats before traffic drops
• How to connect crawl stats with indexing and coverage reports
• A monthly crawl stats review workflow for busy teams
• FAQ
• Sources
• Further reading

<!-- IMAGE-PROMPT cover: 1200x630, DependsIt brand, deep charcoal #121212 background with vibrant mint #22E3B0 accent glow, thin node-network line art, Clash Display style bold heading space on left, General Sans clean labels, subject: crawl stats report cover illustration, flat vector, high contrast, accessible, no photorealistic faces, no text smaller than 24px, no em dash in rendered text, export PNG then cwebp -q 82 to WEBP -->

Where to find the crawl stats report and who should use it

This section covers where to find the crawl stats report and who should use it in the context of crawl stats report. The crawl stats report lives in Search Console under Settings then Crawl stats, with a link to open the full host report. Owners, developers and SEOs all use it for different questions. Owners check overall load, developers check errors and timing, SEOs check whether priority templates get visited. Open it with a date range of at least 90 days for context. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

Google discovers most pages through crawl, not through a single submission. A submission is a hint that asks for a fresh look, but ranking and storage still depend on quality, uniqueness and site trust. That is why steady technical hygiene matters more than any one push. Keep response times low, avoid redirect chains, and return clear status codes. When the crawler can fetch quickly and without loops, each hint carries more weight and uses less of your daily allowance.

Google does not support IndexNow, so plan for two ecosystems. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for eligible types. A practical setup sends product updates to both paths at publish time. One worker prepares the URL list, then one branch pings IndexNow endpoints and another branch queues Google notifications within quota. Coverage improves without double counting.

Internal linking does more for indexing than most teams expect. New URLs that sit four clicks from the home page may wait days for a visit, while URLs linked from a popular category or a recent posts block get visited quickly. Add new products to relevant category pages, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple, stable links help both Google and IndexNow driven crawlers find changes fast.

Thin or duplicated content slows indexing because Google prioritizes pages likely to satisfy searchers. Short product descriptions copied from suppliers, empty category pages and near duplicate articles often sit in Discovered or Crawled without indexing. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing.

In practice, make a short runbook for where to find the crawl stats report and who should use it and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

How to read total crawl requests and host load

This section covers how to read total crawl requests and host load in the context of crawl stats report. Total requests show raw fetch volume while host load shows how hard Google pushes your server. Rising requests with flat sales or flat new content often mean waste on duplicates or filters. Falling requests with new launches often mean blocks, errors or weak signals. Read both together rather than judging one line alone. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

Crawl capacity is often misunderstood. For small sites it rarely limits indexing, but for catalogs with 50000 to 500000 URLs it shapes what gets visited each day. Facets, session parameters, internal search results and duplicate variants can trap crawlers in low value loops. Use robots rules to block filtered views, use canonical tags to consolidate variants, and link best sellers from the home page and category hubs. Fewer dead ends means faster visits to new and updated pages.

• Step 1: Open crawl stats for 90 days and note requests, host load and response mix.
• Step 2: Sample 50 URLs from each spike and label template plus cause.
• Step 3: Fix server, robots or link cause once per template rather than per URL.
• Step 4: Update sitemaps and internal links to point only to keepers.
• Step 5: Recheck stats and coverage after one full crawl cycle.

A 429 means slow down, not try harder. Read the Retry After header when present, then wait with exponential backoff and jitter before retrying. A common pattern waits 2 seconds, then 4, then 8, then 16, with a small random addition to avoid synchronized retries. Cap retries at 4 or 5 and move the URL to a delayed queue after that. Hammering the endpoint during a limit only extends the block and burns log space.

Robots directives and meta tags can silently block indexing. A stray noindex in a template, an X Robots Tag header from a staging config, or a disallow in robots that covers new paths will keep pages out even after successful submission. Audit headers with a fetch tool, render pages as Googlebot, and check the coverage report for Excluded by noindex or Blocked by robots. Fix the template once rather than patching URLs one by one.

Speed and stability raise effective crawl capacity. Compress images, cache HTML at the edge where safe, trim heavy scripts and keep time to first byte steady under load. Monitor 5xx rate, redirect chains and DNS time alongside crawl stats. When the host answers quickly and consistently, Google can do more useful work per minute without raising risk for shoppers and readers.

Use this crawl stats guide with your exports: compare googlebot requests against host load, then check crawl response rates for 404 and 500 growth before you change code. In practice, make a short runbook for how to read total crawl requests and host load and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

To monitor crawling month to month, keep a sheet from crawl stats search console with requests, host load, response mix, and purpose, and note googlebot activity spikes next to deploys. For background on a related report, see crawl budget explained and how limits work which explains how fetch data maps to coverage decisions.

<!-- IMAGE-PROMPT diagram-01: 1600px max, DependsIt brand mint #22E3B0 on charcoal #121212, node-network line art, Clash Display style headings feel with General Sans clean labels, subject: crawl stats report diagram with crawl and queue nodes, flat vector, accessible, no em dash in rendered text -->

How to read crawl responses by type and what they signal

This section covers how to read crawl responses by type and what they signal in the context of crawl stats report. The response breakdown groups fetches into OK, redirect, not found, unauthorized, server error and other. Healthy sites show mostly OK with a small steady share of redirects from normal maintenance. Growing 404 or 5xx shares point to broken links, deploys or origin strain. Each spike deserves a log sample before you change code. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

The Google Indexing API only documents JobPosting and BroadcastEvent pages, which covers job listings and livestream video. Many site owners still test it for product or article URLs, but that use is off label and results vary. Google may process the hint, ignore it, or throttle it. State this plainly to stakeholders. Use the API for eligible content first, and rely on sitemaps, internal links and IndexNow for broad coverage on other page types.

Internal linking does more for indexing than most teams expect. New URLs that sit four clicks from the home page may wait days for a visit, while URLs linked from a popular category or a recent posts block get visited quickly. Add new products to relevant category pages, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple, stable links help both Google and IndexNow driven crawlers find changes fast.

Log analysis shows what crawlers actually did, not what dashboards assume. Group hits by user agent, path template, status code and hour to see waste and priority coverage. Look for Googlebot loops on calendars, filters and search pages, plus spikes after deploys. Share weekly summaries with developers and editors so fixes target the largest waste first. Evidence from logs keeps debates short and actions clear.

Sitemaps remain the backbone of discovery. A clean product or article sitemap lists only canonical, indexable URLs that return 200 and load quickly. Split large catalogs into chunks of 10000 to 40000 URLs, compress with gzip, and reference each chunk from a sitemap index. Update the lastmod field only when content truly changes. Submit the index in Search Console and keep it reachable. A tidy sitemap reduces wasted fetches and leaves room for priority pages.

In practice, make a short runbook for how to read crawl responses by type and what they signal and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

For official details, see crawl documentation which defines how crawling, politeness and host load interact.

How to read crawl purpose discovery refresh and recrawl

This section covers how to read crawl purpose discovery refresh and recrawl in the context of crawl stats report. Purpose splits visits into discovery of new URLs, refresh of known pages and recrawl checks. New sites should see strong discovery, established catalogs should see steady refresh on changing pages. Low discovery with many new URLs points to weak sitemaps or deep orphan pages. Low refresh on prices or stock points to weak update signals. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

Quotas shape every automation decision. Many projects start with about 200 publish requests per day for URL notifications, plus per minute limits that trigger 429 when bursts arrive. Track usage in Cloud Console under APIs and Services, set alerts at 60 percent and 85 percent, and log each publish with timestamp, URL, response code and notification type. When you know your burn rate by hour, you can pace jobs, defer low priority URLs and avoid midnight surprises.

• Step 1: Open crawl stats for 90 days and note requests, host load and response mix.
• Step 2: Sample 50 URLs from each spike and label template plus cause.
• Step 3: Fix server, robots or link cause once per template rather than per URL.
• Step 4: Update sitemaps and internal links to point only to keepers.
• Step 5: Recheck stats and coverage after one full crawl cycle.

Robots directives and meta tags can silently block indexing. A stray noindex in a template, an X Robots Tag header from a staging config, or a disallow in robots that covers new paths will keep pages out even after successful submission. Audit headers with a fetch tool, render pages as Googlebot, and check the coverage report for Excluded by noindex or Blocked by robots. Fix the template once rather than patching URLs one by one.

Google discovers most pages through crawl, not through a single submission. A submission is a hint that asks for a fresh look, but ranking and storage still depend on quality, uniqueness and site trust. That is why steady technical hygiene matters more than any one push. Keep response times low, avoid redirect chains, and return clear status codes. When the crawler can fetch quickly and without loops, each hint carries more weight and uses less of your daily allowance.

Search Console verification is the gate for any Google workflow. The property must be verified with the correct scheme and subdomain, and team access must match the property type. Domain properties and URL prefix properties behave differently, so confirm which one you use before debugging coverage. If you see permission issues, check sharing settings first, then property match, then URL exactness. Most access confusion traces to a missed property detail, not to code.

In practice, make a short runbook for how to read crawl purpose discovery refresh and recrawl and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

To compare link and structure fixes, read how to guide Googlebot crawl rate safely before you edit templates or navigation.

How to spot server problems in crawl stats before traffic drops

This section covers how to spot server problems in crawl stats before traffic drops in the context of crawl stats report. Server trouble appears first as slower downloads, larger average response size or a rising 5xx line. Cross check with uptime, CPU and database slow logs for the same hours. If Google slows host load right after your deploy, treat it as a capacity signal. Fix the bottleneck, then watch for load to recover over several days. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

A 403 usually points to permissions or scope. Confirm the service account email has Owner access in Search Console, confirm the OAuth scope includes the indexing scope, and confirm the JSON key file matches the active key in Cloud Console. Check clock skew on the server, since JWT auth fails when time drifts by more than a few minutes. Rotate keys on a schedule, store them in a secret manager, and never paste private keys into chat tools or shared docs.

Log analysis shows what crawlers actually did, not what dashboards assume. Group hits by user agent, path template, status code and hour to see waste and priority coverage. Look for Googlebot loops on calendars, filters and search pages, plus spikes after deploys. Share weekly summaries with developers and editors so fixes target the largest waste first. Evidence from logs keeps debates short and actions clear.

Crawl capacity is often misunderstood. For small sites it rarely limits indexing, but for catalogs with 50000 to 500000 URLs it shapes what gets visited each day. Facets, session parameters, internal search results and duplicate variants can trap crawlers in low value loops. Use robots rules to block filtered views, use canonical tags to consolidate variants, and link best sellers from the home page and category hubs. Fewer dead ends means faster visits to new and updated pages.

Google does not support IndexNow, so plan for two ecosystems. IndexNow notifies Bing, Yandex, Naver, Seznam and other partners that share the protocol, while Google relies on sitemaps, Search Console inspection and the Indexing API for eligible types. A practical setup sends product updates to both paths at publish time. One worker prepares the URL list, then one branch pings IndexNow endpoints and another branch queues Google notifications within quota. Coverage improves without double counting.

In practice, make a short runbook for how to spot server problems in crawl stats before traffic drops and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

import re
from collections import Counter
pat = re.compile(r'"GET (\S+) HTTP.*?" (\d{3})')
counts = Counter()
for line in open('access.log', encoding='utf-8', errors='ignore'):
    m = pat.search(line)
    if m and 'Googlebot' in line:
        counts[m.group(2) + ' ' + m.group(1).split('?')[0][:60]] += 1
for k, v in counts.most_common(20):
    print(v, k)

How to connect crawl stats with indexing and coverage reports

This section covers how to connect crawl stats with indexing and coverage reports in the context of crawl stats report. Crawl without indexing means fetch succeeded but selection did not. Join crawl stats with the Pages indexing report and URL Inspection. If fetches rise while Valid stays flat and Discovered grows, focus on quality, canonicals and internal demand. If fetches fall while Valid falls, focus on blocks, errors and sitemap health. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

Canonical tags decide which URL keeps the indexing credit. If variants with color, size or tracking parameters lack a canonical, Google may pick a different URL or delay indexing while it compares duplicates. Point each variant to the preferred canonical, keep the canonical self referencing on the main URL, and make sure sitemaps list only canonicals. For translated or regional pages, add hreflang and keep each locale self consistent. Clean signals shorten the decision time.

• Step 1: Open crawl stats for 90 days and note requests, host load and response mix.
• Step 2: Sample 50 URLs from each spike and label template plus cause.
• Step 3: Fix server, robots or link cause once per template rather than per URL.
• Step 4: Update sitemaps and internal links to point only to keepers.
• Step 5: Recheck stats and coverage after one full crawl cycle.

Google discovers most pages through crawl, not through a single submission. A submission is a hint that asks for a fresh look, but ranking and storage still depend on quality, uniqueness and site trust. That is why steady technical hygiene matters more than any one push. Keep response times low, avoid redirect chains, and return clear status codes. When the crawler can fetch quickly and without loops, each hint carries more weight and uses less of your daily allowance.

The Google Indexing API only documents JobPosting and BroadcastEvent pages, which covers job listings and livestream video. Many site owners still test it for product or article URLs, but that use is off label and results vary. Google may process the hint, ignore it, or throttle it. State this plainly to stakeholders. Use the API for eligible content first, and rely on sitemaps, internal links and IndexNow for broad coverage on other page types.

A 429 means slow down, not try harder. Read the Retry After header when present, then wait with exponential backoff and jitter before retrying. A common pattern waits 2 seconds, then 4, then 8, then 16, with a small random addition to avoid synchronized retries. Cap retries at 4 or 5 and move the URL to a delayed queue after that. Hammering the endpoint during a limit only extends the block and burns log space.

In practice, make a short runbook for how to connect crawl stats with indexing and coverage reports and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

<!-- IMAGE-PROMPT workflow-02: 1600px max, DependsIt brand mint #22E3B0 on charcoal #121212 or white, node-network line art, Clash Display style headings feel with General Sans clean labels, subject: crawl stats report workflow with review and audit steps, flat vector, accessible, no em dash in rendered text -->

A monthly crawl stats review workflow for busy teams

This section covers a monthly crawl stats review workflow for busy teams in the context of crawl stats report. A short monthly routine beats occasional deep dives. Export requests, host load, response mix and purpose, note top growing file types and response codes, sample 50 URLs from each spike and assign one owner per fix. Keep a one page log with dates, actions and before and after charts for the next review. We keep the advice practical for owners without a large team. Each check below uses Search Console, logs and a small crawl you can run today. The goal is steady progress you can see in coverage, not a one time spike. Keep notes on what you change and when, so indexing movement links clearly to specific fixes and dates.

Thin or duplicated content slows indexing because Google prioritizes pages likely to satisfy searchers. Short product descriptions copied from suppliers, empty category pages and near duplicate articles often sit in Discovered or Crawled without indexing. Add specific details such as dimensions, materials, compatibility, usage steps and original photos. Consolidate near duplicates into one strong page with redirects. Better content earns more frequent revisits and steadier indexing.

Crawl capacity is often misunderstood. For small sites it rarely limits indexing, but for catalogs with 50000 to 500000 URLs it shapes what gets visited each day. Facets, session parameters, internal search results and duplicate variants can trap crawlers in low value loops. Use robots rules to block filtered views, use canonical tags to consolidate variants, and link best sellers from the home page and category hubs. Fewer dead ends means faster visits to new and updated pages.

Quotas shape every automation decision. Many projects start with about 200 publish requests per day for URL notifications, plus per minute limits that trigger 429 when bursts arrive. Track usage in Cloud Console under APIs and Services, set alerts at 60 percent and 85 percent, and log each publish with timestamp, URL, response code and notification type. When you know your burn rate by hour, you can pace jobs, defer low priority URLs and avoid midnight surprises.

Internal linking does more for indexing than most teams expect. New URLs that sit four clicks from the home page may wait days for a visit, while URLs linked from a popular category or a recent posts block get visited quickly. Add new products to relevant category pages, link related items, and keep pagination crawlable with plain anchors. Avoid loading key links only through scripts that require clicks. Simple, stable links help both Google and IndexNow driven crawlers find changes fast.

In practice, make a short runbook for a monthly crawl stats review workflow for busy teams and review it after each deploy. List who owns Search Console, where logs live, which sitemap covers the URLs and what alert fires first. Test with a small sample before wider rollout. Record status codes and timestamps so patterns appear without guesswork. If errors rise, pause, fix the root cause, then resume at half pace. Steady documented pacing beats rushing to catch up in one burst. Share the runbook with developers and editors so ownership stays clear through staff changes and seasonal peaks.

FAQ

Where is the crawl stats report?

Open Search Console, go to Settings, then Crawl stats, then Open report. Use a 90 day window to see trends. Check total requests, host load, responses and purpose before judging any single spike. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory. Small consistent records make the next incident faster to resolve and easier to explain.

What is a healthy response mix?

Mostly 200 OK with a small steady share of 301 and 304 from normal maintenance. Teams that track crawl stats purposes such as discovery, refresh, and recrawl can tell new page discovery apart from routine refresh. Rising 404 or 500 lines deserve log samples. Track the mix monthly so gradual drift stands out early. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory. Small consistent records make the next incident faster to resolve and easier to explain.

Why does discovery stay low after launches?

New URLs may sit deep without links, sitemaps may list non canonical or blocked URLs, or robots may block paths. Fix sitemap accuracy, add hub links and test a sample with URL Inspection. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory. Small consistent records make the next incident faster to resolve and easier to explain.

Do crawl stats show index status?

No. They show fetch activity only. Pair them with the Pages indexing report and inspection results to see whether fetched pages were stored, consolidated or left out. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory. Small consistent records make the next incident faster to resolve and easier to explain.

How often should I review crawl stats?

Monthly for stable sites, weekly during migrations, launches or outage recovery. For deeper crawl stats insights, annotate each review with template fixes so purpose shifts link clearly to actions. Keep notes with dates so you can link movement to deploys, fixes and content releases. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory. Small consistent records make the next incident faster to resolve and easier to explain.

Can I export crawl stats data?

Use the report export and the URL Inspection API plus server logs for detail. Export plus logs lets you read crawl data per template instead of guessing from charts alone. Keep a sheet with requests, host load, response mix and purpose by week for easy comparison over quarters. Keep a short log of what you checked and when, so the next review starts from evidence rather than memory. Small consistent records make the next incident faster to resolve and easier to explain.

Sources

• Official crawl documentation
• Search Console help reference

Further reading

• why pages stay discovered but not indexed
• fixes for crawled but not indexed pages
• how server errors drop pages from the index
• how to submit sitemaps in Search Console