Your llms.txt Isn't Being Ignored — Your Analytics Just Can't See It

Shibin Das July 22, 2026

There’s a claim I keep running into, in blog posts and in threads and in the confident voice of someone who has Just Checked:

“I added an llms.txt. Nothing reads it. Zero hits. The whole idea is snake oil.”

I want to take that apart — not to defend llms.txt, which may or may not deserve defending, but because the evidence being offered is almost always broken. The person saw a zero. The zero was real. The zero meant nothing.

The mistake isn’t the conclusion. It’s the instrument.

What llms.txt Is, Briefly

llms.txt is a proposed convention: a Markdown file at the root of your domain (/llms.txt) that gives large language models a curated, low-noise map of your site — what the pages are, which ones matter, where the canonical docs live. Think robots.txt’s intent, but written for a reader that wants prose instead of crawl rules.

Whether AI systems actually consume it at scale is a live, unsettled debate — and an honest one. This post is not that debate. This post is about the much smaller, much more common error people make while trying to answer that debate for their own site.

They open their analytics dashboard, search for /llms.txt, find nothing, and file it under “confirmed useless.”

That dashboard was never going to show them anything. Not because the file wasn’t fetched — because of how the dashboard works.

How Client-Side Analytics Actually Records a Hit

Google Analytics, Plausible, Umami, Fathom, Matomo-in-JS-mode — the whole client-side family — all work the same way underneath. They do not watch your server. They watch browsers.

The mechanism is a snippet:

<script defer src="/script.js" data-website-id="..."></script>

The chain of events that produces one recorded “pageview” is:

A browser requests an HTML page.
The browser parses the HTML and executes the JavaScript in it.
That script runs, reads some document and navigator state, and fires a beacon back to the analytics collector.
The collector records the hit.

Every step after the first is a browser doing browser things. Take any one of them away and the hit never exists as far as the dashboard is concerned.

Now look at what happens when an AI crawler — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, or a plain curl from someone’s agent script — fetches your llms.txt:

It issues a raw HTTP GET /llms.txt.
It reads the bytes.
It leaves.

No browser. No JavaScript engine. No beacon. The fetch happened, the bytes went out over the wire, your server logged a 200 — and your analytics recorded nothing, because nothing in that transaction ever ran the code that analytics depends on.

And It’s Worse Than That: A .txt File Can’t Even Hold the Snippet

Here’s the part that should end the argument on its own.

Client-side analytics needs its <script> tag to be in the document being viewed. That’s fine for index.html. But llms.txt is plaintext. There is no <head>. There is nowhere to put the snippet. You could not instrument that file with Google Analytics if you tried.

So even in the fantasy scenario where a bot ran a full JavaScript engine against your text file — which it doesn’t — there would still be no snippet on the page to fire. The file is invisible to client-side analytics by construction, independent of who or what is fetching it.

Concluding “bots don’t read my llms.txt” from a client-side dashboard is like concluding your mail is never delivered because you keep checking the microwave. The tool isn’t giving you a weak signal. It is not connected to the thing you’re asking about.

Where the Answer Actually Lives

The transaction that analytics missed was recorded — just somewhere else. It’s in your server-side access logs: Nginx, Apache, Caddy, or your edge provider’s log stream.

This site is a good example. It loads no analytics script, sets no cookies, fires no beacon of any kind — by design. If I relied on client-side tracking, I would have zero visibility into anything, human or bot. The only traffic picture I get is the aggregate, server-side one Vercel derives from request logs. And that’s exactly the layer where a bot fetching a text file shows up.

If you have raw access logs, the check is one line:

grep '/llms.txt' access.log | awk '{print $1, $12}' | sort | uniq -c | sort -rn

Or, filtering for the user agents that actually matter:

grep '/llms.txt' access.log \
  | grep -iE 'GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Google-Extended|Bytespider|CCBot' \
  | wc -l

On an edge platform without shell access to logs, the same question is a filter in the dashboard: request path contains /llms.txt, grouped by user agent. Cloudflare, Fastly, and friends all expose this. It’s a different tool than your marketing analytics, and that’s the whole point — it’s a tool that sits at the layer where the request lives.

Here’s a rough sense of what the two sources report for the same file over the same week:

Source	Sees HTML pageviews	Sees `/llms.txt` bot fetches
Client-side analytics (GA, Plausible, Umami…)	✅	❌
Server / edge access logs	✅	✅

The bot column is the entire question. Only one row can answer it.

Say the plaintext problem is fully solved — you’re looking at server-side numbers now, not a JS beacon. You can still land on a phantom zero, for a completely different reason: most analytics products filter bots out on purpose, and treat it as a selling point.

The JavaScript gap is accidental — a side effect of how beacons work. This one is deliberate. These tools exist to answer “how many humans visited?”, so a fetch that announces itself as GPTBot is noise to be removed, not signal to be counted. They match the incoming user agent against a known-bot list — projects like isbot or a “beep boop” regex — and quietly drop anything that matches before it ever reaches a chart.

Umami is the tidy example of both gaps stacked in one tool. It’s a client-side beacon (so a non-browser fetch never fires it — blind spot one), and it runs incoming traffic through bot detection to keep known crawlers out of the stats (blind spot two). Either mechanism alone is enough to guarantee your llms.txt bot fetches never appear. Together they make it structurally impossible.

And the trap is sharper here, because a bot-filtering tool looks like it’s at the right layer. It processes real requests. It just made counting bots someone else’s job — yours — and told you so in the docs you didn’t read.

This is why the reliable source is raw access logs, not a polished analytics layer sitting on top of them. The moment a tool’s headline feature is “we filter out bot traffic for you,” it has disqualified itself from answering “did a bot fetch this?” The filter you’re fighting is the one they’re proud of.

The Honest Caveat (Because the Zero Might Be Real)

I said the zero meant nothing. I need to be precise: the zero from client-side analytics means nothing. A zero from your server logs means something.

If you grep your access logs and genuinely find no bot ever requested /llms.txt, that’s a real, valid finding — and a reasonable input to the “is this worth maintaining?” decision. Absence of evidence is finally worth something once you’re measuring at the right layer.

Two more honesty notes, so nobody swings too far the other way:

A fetch is not a use. A logged 200 for GPTBot proves the file was retrieved. It does not prove the content shaped any model’s answer. That’s a genuinely hard thing to measure, and server logs don’t measure it either. Don’t upgrade “it was fetched” into “it worked.”
User agents lie. Anyone can send User-Agent: ClaudeBot. Log analysis tells you what knocked, not always who. For most purposes that’s fine; just don’t build a fortress on a header a client controls.

The General Shape of the Mistake

Strip away llms.txt and this is an old, reusable error: reaching for the instrument you have instead of the one the question requires, then trusting its silence.

Client-side analytics answers “how are humans-with-browsers moving through my HTML?” It’s genuinely good at that. It is definitionally silent about raw fetches of non-HTML resources by clients that don’t run JavaScript — bots, scrapers, agents, curl, RSS readers, uptime monitors, half your API traffic. Ask it about any of those and it will confidently return zero, because zero is the only number it can produce. The silence isn’t data. It’s the sound of a microphone that was never plugged into the room.

So before you conclude anything reads — or doesn’t read — your llms.txt:

Don’t ask client-side analytics. It cannot see plaintext fetches by non-browsers, and a .txt file can’t even carry its snippet.
Don’t trust a bot-filtering analytics tool either. If removing crawlers is its headline feature, it can’t tell you crawlers showed up.
Ask your raw server or edge access logs. Filter by path, group by user agent, and make sure bots are included, not stripped.
Then, and only then, believe the zero — or the number you actually find.

The file might be pointless. But you don’t get to say so from a dashboard that was structurally incapable of counting the thing you’re arguing about.

Engineering