Log File Analysis: The Hidden Crawl Budget Audit Every B2B SaaS Needs in 2026
TL;DR: Log file analysis B2B SaaS teams can run on their own infrastructure is the only way to see what real crawlers — including Googlebot and the new wave of AI bots — actually do on your site, and it routinely uncovers crawl waste, orphan URLs and indexing issues that no third-party tool can simulate.
For most B2B SaaS sites, technical SEO work is reactive: a tool flags a 404, someone fixes it, and the next audit is scheduled for next quarter. Log file analysis breaks that cycle. Your server access logs are a faithful record of every request a crawler (or a user) has made in the period you keep them. They show, in order, with full URLs and status codes, exactly which pages Googlebot, Bingbot, GPTBot and the rest are touching, which ones they are ignoring, and which ones they are hammering that you wish they would not. The value of log file analysis B2B SaaS teams unlock is not a single audit, but a ground-truth dataset that you can query again and again.
What Log File Analysis B2B SaaS Teams Actually Discover
A standard technical audit tells you what a tool *thinks* your site looks like. Your log files tell you what crawlers *actually did*. The difference is large, and it is the reason log file analysis is a different discipline from technical crawling rather than a subset of it.
Three things tend to surface quickly. First, there is almost always a long tail of URLs being requested that nobody on the SEO team knows about — legacy paths, internal search results, paginated documentation, tracking parameters, locale variants. Second, the relative crawl weight across sections of the site is rarely what you would expect from looking at the sitemap. Third, you can see clearly when Googlebot has stopped visiting a section that you assumed was being indexed.
The single most important thing log files give you is ground truth: what real crawlers actually do, not what a tool guesses they would do. A surprising number of "the index must be broken" tickets in a SaaS backlog close themselves once you look at the logs and see that the affected pages have not been crawled in months.
How to Get and Read Your B2B SaaS Log Files
Before you can analyse anything, you need the raw data. The location depends on your stack. On a self-hosted nginx or Apache setup, you are looking at `access.log` in Combined Log Format or Common Log Format.
On a managed platform — Vercel, Netlify, Heroku, Render, AWS Amplify — log delivery is usually a configuration step rather than a file path. Most production SaaS sites sit behind a CDN such as Cloudflare, Fastly, CloudFront or Akamai, and the useful logs are the edge logs there, not the origin logs.
The fields that matter for SEO work are: timestamp, client IP, HTTP method, full request URL (including query string), status code, bytes sent, referer, and user agent. JSON-formatted logs are increasingly common and are easier to pipe into a warehouse. Pull at least 30 days of raw logs before you start any analysis — anything shorter will be skewed by crawl spikes around releases or outages.
Once you have the data, the analysis layer depends on volume. For a small SaaS site, a desktop tool such as the Screaming Frog Log File Analyser is enough. For anything larger, you are usually better off loading the logs into Splunk, ELK, Datadog, BigQuery or a similar platform and querying them directly. The data sources you can cross-reference against are limited, and choosing the right one matters:
| Data Source | What It Shows | What It Misses | Best For |
|---|---|---|---|
| Server log files | Every real request from every bot and user, with timestamp, status code, user agent and full URL | Your infrastructure view only; no SERP or backlink context | Crawl budget audits, orphan page discovery, real bot behaviour |
| Google Search Console (Crawl Stats, URL Inspection) | Aggregated Googlebot activity and sampled URL data for a verified property | Sampled, delayed, limited to Google, no per-request detail | Quick trend checks, coverage issues, manual indexing |
| Third-party crawlers (Screaming Frog, Sitebulb, Ahrefs) | Simulated crawl of your site, including links, status, render and on-page data | Not Googlebot — the crawl path is synthetic | Technical SEO audits, internal link analysis, on-page checks |
| CDN / edge logs (Cloudflare, Fastly, Akamai) | Real edge traffic including bots, often enriched with country, device and threat score | Aggregation overhead, bot detection varies, log push often required | High-traffic sites, bot-blocking decisions, security analysis |
A useful first query is simply "URLs requested by Googlebot more than N times in the last 30 days, sorted descending". The output is almost always a conversation starter.
Crawl Budget Optimisation via Log File Analysis in B2B SaaS
Crawl budget is the colloquial term for how much crawling a search engine will spend on your site in a given window. It is governed by two things: the crawl rate limit (how many requests your server can comfortably serve without slowing down) and crawl demand (how much of your content search engines actually want to index). For B2B SaaS sites, the second factor is usually the binding constraint, which means a small, well-structured site with thin or duplicate sections will have its crawl budget wasted on the wrong pages.
The waste shows up clearly in logs. Look for URL patterns that are being requested at high frequency but that you do not actually want indexed: integration directory facets, internal search results, paginated documentation beyond the first few pages, calendar-style archive pages, A/B test variants, marketing tracking parameters. Then look at the inverse — important URLs in your product, pricing, comparison or documentation area that are barely being touched. The right question to ask of every recurring high-frequency request is "does this URL deserve the attention it is getting?" If the answer is no, you have three options: consolidate it behind a canonical, noindex it, or block it in robots.txt.
Crawl budget optimisation through logs is also the cleanest way to measure the impact of a fix. If you tighten robots.txt to block a facet pattern, you should see the request volume for that pattern drop in the days that follow, and ideally you should see the crawl reallocate to higher-value pages in the same period. Without logs, you would be guessing.
Orphan Pages, Crawl Traps and Faceted Navigation in SaaS Logs
Log files redefine what "orphan page" means in a useful way. The classic definition is a page with no internal links pointing to it, which a tool like Screaming Frog will surface by walking the link graph. A log-driven definition is stronger: an orphan page is one that Googlebot is reaching *despite* there being no internal link to it, which means there is an external or sitemap path that has put it on Google's map but your own navigation has not.
This is common in B2B SaaS. Think old launch posts, partner landing pages that no longer exist on the marketing site but still appear in PDF case studies, programmatic SEO pages generated and forgotten, and integration pages that were once hand-built and are now generated through a different template.
Crawl traps are the related problem: URL patterns that Googlebot gets stuck inside, generating ever more requests for essentially the same content. The classic SaaS example is an integrations directory with facet combinations (`/integrations?category=analytics&sort=popular&page=4&vendor=snowflake`), an internal search endpoint exposed without noindex, or a documentation tree with infinite date-based pagination. Every recurring high-frequency request on a low-value URL is a candidate for canonicalisation, noindex, or robots blocking — and logs are how you find them. For more on how SaaS teams approach orphan and thin page cleanup, our insights library has a few related pieces.
AI Crawlers in B2B SaaS Log File Analysis
A noticeable shift in log files over the last couple of years has been the volume and variety of AI crawlers. There are broadly two categories. The first is training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, which feeds most model training pipelines), Google-Extended (a separate user agent for Gemini training), Applebot-Extended, and Bytespider (TikTok/ByteDance).
The second is retrieval or search crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, and a handful of others. They each obey robots.txt independently of Googlebot, and they each make their own decisions about how often to fetch.
The strategic decision is yours. Some SaaS companies want to be cited by ChatGPT, Perplexity and Gemini, in which case blocking the relevant bots is counterproductive. Others, particularly those with significant paywalled or proprietary content, prefer to block the training crawlers and selectively allow the retrieval ones. The logs are how you find out what is actually happening today, before you make that policy. AI crawlers are now a separate dimension of crawl budget, and they need a deliberate robots.txt policy rather than being treated as a curiosity.
A small but important caveat: user agent strings are not trustworthy on their own. If a request claims to be Googlebot, verify it through reverse DNS on the client IP. The same applies, in principle, to other named crawlers, though their verification processes are less mature. If you see a request labelled as GPTBot coming from an IP range that OpenAI does not publish, treat it with suspicion.
Building a Repeatable Log File Analysis B2B SaaS Workflow
A one-off log dive gives you a snapshot. The real value is in turning it into a recurring discipline. For most B2B SaaS sites, monthly is the right cadence, stepping up to weekly during a major migration, a re-platforming, or in the immediate aftermath of a Google update. Quarterly is the minimum if resources are tight.
The workflow itself is straightforward but only if you keep it disciplined. Step one: extract the raw logs for the period and land them in a queryable store. Step two: clean them, deduplicate, normalise query strings if you want to group by pattern.
Step three: segment by user agent, separating Googlebot variants, Bingbot, the AI crawlers, known SEO tool bots and unverified traffic. Step four: join with your URL inventory, ideally one that has been enriched with section, template type and business priority. Step five: flag patterns — high frequency on low-value URLs, low frequency on high-value URLs, unexpected user agents, large volumes of 4xx and 5xx on important paths.
Step six: turn the flags into a prioritised backlog with clear owners.
Common mistakes worth naming. People often treat 304 (Not Modified) responses as if they do not count, but they do — every 304 is a request Googlebot still made. People frequently forget to factor in soft 404s, which return a 200 status with a "not found" page body. People also tend to underweight the slow decay of crawl on important sections, which only becomes visible when you look at trends rather than a single month. A one-off log dive is a snapshot, and the real return comes from doing it the same way every month and watching the trend lines move.
If you need a partner to design or run this workflow with you, our services cover everything from initial audit design to monthly monitoring, and you can contact us to scope an engagement.
Frequently Asked Questions
How often should a B2B SaaS company run log file analysis?
For most teams, monthly is the right cadence. Move to weekly during a site migration, a major release, or in the four to six weeks following a Google core update, when crawl patterns can shift suddenly. Quarterly is the floor — anything less frequent and you will miss short-lived crawl anomalies that explain organic traffic changes.
What is the difference between log file analysis and a standard technical SEO crawl?
A standard crawl simulates what a bot would do by walking your internal link graph. It is a model. Log file analysis is the record of what real bots — primarily Googlebot, plus the AI crawlers — actually requested, in what order, and with what response codes.
Crawls are great for finding on-page and link-graph issues. Log files are the only reliable way to understand crawl budget, real bot behaviour and the orphan-page situation from the crawler's point of view.
Can log file analysis help recover from a Google algorithm update or a site migration?
Indirectly, yes. After a migration, logs are the fastest way to confirm that Googlebot is finding the new URL templates and that the redirects are being followed as intended. After an algorithm update, logs will show you whether crawl on the affected sections has dropped, and whether the surviving crawl is concentrated on the URLs you would want to win. They do not directly tell you why rankings moved, but they rule out crawl-related explanations quickly.
How do I verify that a request in my logs is genuinely from Googlebot or GPTBot?
For Googlebot, the supported process is reverse DNS on the client IP: resolve the IP, confirm the resulting host ends in `googlebot.com` or `google.com`, then forward-resolve it back to the IP. If the round trip matches, the request is genuine. For other named crawlers, check the operator's published IP ranges and verification documentation; the process is less formalised, so treat user agent strings as a hint rather than proof.
What is the minimum viable log file analysis setup for an early-stage SaaS site?
A month's worth of access logs in their raw format, opened in the Screaming Frog Log File Analyser (or a similar lightweight tool), filtered to Googlebot, sorted by URL. That single view, refreshed monthly, will surface most of the high-value findings you would otherwise miss: the waste, the orphans and the unvisited important pages. As the site grows, you can graduate to a proper log pipeline and warehouse.
Key Takeaways
- Ground truth matters: Log file analysis B2B SaaS teams run on their own infrastructure is the only dataset that records what real crawlers did, not what a third-party tool would simulate.
- Pull the raw data first: At least 30 days of unprocessed access logs, in the right format, before you commit to any specific analysis tool.
- Ask the right question of every URL pattern: Does this URL deserve the crawl attention it is getting, and does the inverse hold for my important pages?
- Crawl budget is mostly a demand problem: The fix is rarely "ask Google to crawl more" and almost always "stop wasting the crawl you have on low-value URL patterns".
- AI crawlers are a separate budget line: GPTBot, ClaudeBot, PerplexityBot and the rest need their own robots.txt policy and their own reporting line.
- Verify before you trust: Reverse DNS for Googlebot, published IP ranges for others — user agent strings are not proof.
- Repeatability beats one-off audits: A monthly log file analysis B2B SaaS programme turns individual findings into a trend you can act on quarter after quarter.
If your team would like support turning log file analysis B2B SaaS into a recurring, decision-ready programme, IvanHub works with London-based and remote teams on this kind of SEO infrastructure.
KEY TAKEAWAYS
- Ground truth matters: Log file analysis B2B SaaS teams run on their own infrastructure is the only dataset that records what real crawlers did, not what a third-party tool would simulate.
- Pull the raw data first: At least 30 days of unprocessed access logs, in the right format, before you commit to any specific analysis tool.
- Ask the right question of every URL pattern: Does this URL deserve the crawl attention it is getting, and does the inverse hold for my important pages?
- Crawl budget is mostly a demand problem: The fix is rarely "ask Google to crawl more" and almost always "stop wasting the crawl you have on low-value URL patterns".
- AI crawlers are a separate budget line: GPTBot, ClaudeBot, PerplexityBot and the rest need their own robots.txt policy and their own reporting line.
- Verify before you trust: Reverse DNS for Googlebot, published IP ranges for others — user agent strings are not proof.
Frequently asked questions
The Compounding Letter
One short note a month. Growth lessons from inside real engagements. No fluff.
MORE INSIGHTS
Next step