Last week I ran Google's agentic browsing audit against our own site, mostly out of curiosity. We scored 2 out of 3. The check we failed was llms.txt.
The annoying part: we had an llms.txt. If we'd had no file at all, Lighthouse would have marked that check not applicable and we'd have walked away clean. Instead our file used plain-text bullets rather than Markdown links, so the audit read it, decided it was malformed, and handed us a zero.
That's the whole llms.txt situation in miniature. Everyone is arguing about whether the file does anything. Almost nobody has checked whether theirs is even valid.
What llms.txt is, and what it isn't
llms.txt is a Markdown file at your domain root. It's a curated map: a title, a short summary of what the site is, then lists of links to the pages you'd want a language model to read. The proposal comes from Jeremy Howard, and llmstxt.org is still the reference.
It is not robots.txt. There's no allow, no deny, no access control of any kind. If you want to gate AI crawlers, that's a robots.txt and WAF conversation, not this file.
It is not a sitemap either. A sitemap is exhaustive and written for machines that crawl. llms.txt is short and opinionated, and you decide what goes in it.
And it is not a ranking factor. Google is unusually blunt on this one. From its AI optimization guide, in the mythbusting section: "You don't need to create new machine readable files... as Google Search itself doesn't use them." In case that left room for hope, the same page adds that doing so "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
John Mueller went further. Search Engine Journal reported him in June 2026 calling llms.txt "purely speculative for now," with a rule of thumb I like: when an AI platform that brings you clients complains that it needs the file, that's when it's worth your time.
The case against llms.txt
The best evidence here is the Ahrefs study published 15 June 2026. They pulled server logs across 137,210 domains for the month of May and looked at who requested llms.txt.
| What they measured | Result |
|---|---|
| Domains publishing an llms.txt | 28% (their sample skews high) |
| Files that got zero traffic in May | 97% |
| Requests that came from bots at all | 96% |
| Fetches attributable to named AI tools | 19.5% |
| Top AI fetcher | GPTBot, 4.51% of requests |
| Probing for llms.txt on sites without one | none |
That last row is the one that should bother you. If ChatGPT or Perplexity were quietly checking for llms.txt as part of how they read the web, you'd see requests hitting 404s on sites that never published one. Ahrefs saw zero. Nobody is looking for the file. It only gets fetched when something already knows it's there.
Their conclusion, verbatim: an llms.txt file is "largely decoration."
For AI Overviews, ChatGPT, and Perplexity visibility, I think that's correct, and I haven't seen anything credible that contradicts it. If someone sold you llms.txt as a path to getting cited in AI answers, you were sold a story. What does move that needle is a longer and much less convenient conversation, and we get into it in what actually gets you cited in LLM answers.
The case for llms.txt
Now go look at who publishes one.
Anthropic does, on platform.claude.com. OpenAI does, on developers.openai.com. Cloudflare, Vercel, and Supabase all do. Google's Gemini API documentation publishes one too, tucked at ai.google.dev/gemini-api/docs/ rather than the domain root, which is its own small comedy given the mythbusting page above.
Every major AI lab ships an llms.txt for their developer docs. That isn't marketing.
Howard is honest about why, in the v2 update to the spec from August 2026: llms.txt is "used most heavily for software documentation, where coding agents follow them." That's the real use case. Not search visibility. A coding agent pointed at a docs site, reading a curated index instead of guessing at your navigation. Yoast's free tier, Mintlify, GitBook, AIOSEO, and Wix all generate the file automatically now, which is why publish rates keep climbing while the traffic numbers don't.
So the honest framing: Google Search ignores llms.txt, Google's Lighthouse audits its format, and Google's Gemini docs publish one. It's a developer-documentation convention that got mistaken for an SEO tactic. If your product has an API, an SDK, or docs that engineers read, the file earns its keep for the same reason good technical documentation does. If you sell dental implants, it does approximately nothing.
Two more reasons it's cheap to keep around: Vercel's Agent Readability Spec lists llms.txt as a site-level requirement, and Lighthouse now checks it. Which brings us to the part people get wrong.
A broken llms.txt is worse than no llms.txt
Lighthouse 13.3.0 shipped on 7 May 2026 with a new category called agentic browsing. It's in Chrome 150 DevTools and PageSpeed Insights, and it's scored as a fraction rather than out of 100. Google is upfront that the category is informational for now, not a benchmark. On standard Chrome, three of its six audits are WebMCP-related and come back N/A, so in practice you're taking a three-question test: accessibility tree, layout stability, llms.txt.
The llms.txt audit checks format, not presence:
| Condition | Score |
|---|---|
| No file (404) | N/A, treated as optional, no penalty |
| File present, passes format checks | Pass |
| File present, fails any format check | 0 |
The format rules are not demanding. The file needs a line that looks like an H1, so `# Your Company` at the top. It needs at least one real Markdown link, which is the rule that got us: `- About us: https://example.com/about-us/` reads like a link to a human and is nothing to a parser, so it has to be `- About us`. And the whole file has to be at least 50 characters, so a lone heading fails.
Three rules, and missing one costs you more than never writing the file. When I ran that same audit sweep, stripe.com came back 3/3, shopify.com 2/3 on ARIA and button-name violations, and airbnb.com 1/2. If Shopify and Airbnb are failing agentic browsing checks, you probably are too. I've written up the full audit and what each check means in what Lighthouse's agentic browsing audit actually checks.
Ship it correctly in 30 minutes
Run the audit first so you know where you stand:
npx lighthouse https://yoursite.com --only-categories=agentic-browsing
Then write the file by hand. Not generated, not scraped, not a zip you bought. Pick the 15 to 30 URLs you'd hand a new engineer or a serious prospect, and write it out:
# Attract Group
> Custom software development company building healthcare, logistics, and > fintech platforms for clients in the US and EU.
Services
- Agent readiness: making sites legible and operable for AI agents.
- AI chatbot development: scope, team structure, pricing.
Reference
- Case studies: shipped products with outcomes.
- Blog: technical writing on the above.
Rules I'd hold to:
- Every entry is a real Markdown link with a colon and a one-line description. That description is for the model, not for a human skimming, so say what's on the page instead of selling it.
- Point at pages that render useful text without JavaScript. No major AI crawler executes JS (Vercel's crawler data, November 2024), so a link to a client-rendered page is a link to an empty div.
- Keep it to a few dozen links at most. The point is curation. List everything and you've written a sitemap with extra steps.
- Validate it. llmstxtvalidator.dev and llms-txt.io/validator both work, and re-running the Lighthouse command above is the check that counts. Skip llmstxt.firecrawl.dev, it's unmaintained.
- Put it at `/llms.txt` and wire it into whatever process updates your sitemap, or it'll be stale within a quarter.
One optional extra, if you have real docs. The v2 spec adds Markdown companion pages, so `/docs/thing.html` gets a `/docs/thing.html.md` twin, advertised with `rel="alternate" type="text/markdown"`. That's more useful than the index file itself and almost nobody does it. Cloudflare found only 3.9% of the top 200,000 domains serve Markdown when asked for it.
Don't buy an llms.txt package
There's a tier of vendors selling protocol-file bundles at $99 to $399 a month, marketed as needing no developer. At the other end, WebFX publishes a generative engine optimization price starting at $3,000 a month. Either way you're paying rent on a text file that takes half an hour to write and changes twice a year.
Worse, it's the wrong end of the list. Rank agent-readiness work by how much evidence supports it and llms.txt lands around eighth. Ahead of it: your accessibility tree and semantic HTML, which is the representation agents genuinely read; server-rendered HTML, because crawlers don't run your JavaScript; layout stability, because an agent taking screenshots gets confused when your buttons move; and per-agent robots and WAF rules, because there's a real difference between a search bot you need to allow and a training bot you can refuse. That last distinction trips up almost everyone, and it's covered properly in our guide to AI crawlers.
llms.txt is the easy item. Doing the easy item and skipping the hard ones is how you end up with a valid llms.txt on a site no agent can use.
So, does it work?
| If you are | Then llms.txt |
|---|---|
| A docs, API, or dev-tool company | Worth doing properly. Coding agents read it. |
| A content or ecommerce site | Cheap insurance. Won't move AI visibility. |
| Already publishing one | Go validate it today. Broken is worse than absent. |
| Being sold it as an AI SEO service | Walk away. |
The other Mueller line from that June interview is the one that sticks with me: "Don't block agents. I think that hurdle will be the biggest, for most sites." Access beats metadata. A site an agent can read and act on will outperform a site with a beautiful llms.txt behind a JavaScript wall, every time.
So write the file, get the format right, and go spend the rest of the quarter on the checks that carry weight.
If you'd like to know where your own site stands, we run a free GEO audit: real Lighthouse numbers, what the AI engines currently say about you, crawler access, and a ranked list of what to fix first. Details here.
