llms.txt vs robots.txt: why block AI bots and still publish an llms.txt?

by Scott Kingin AEOon Posted on

I found 13 sites that wrote a file for AI tools and then turned ChatGPT away at the door.

I asked ChatGPT to open 132 B2B software sites that a bot test had flagged. 34 blocked it on both Oct 29 and Oct 30, 2026. 13 of those 34 publish an llms.txt.

My read: two teams own two files, and nobody compares them. I call it the welcome-mat problem. One team rolls out the mat. Another team locks the door.

This piece goes deeper on one section of my crawl of B2B software websites. Call it GEO or AEO, the file only matters if the bot can get in.

What did I find when I checked the blocked sites?

The base is small. These 34 sites come from the 132 a simulated-bot test flagged, so treat the counts as counts. 13 of 34 says what happened in this group. It isn’t a rate for B2B software.

For context, llms.txt is common in this cohort. 1,041 of the 3,233 sites I could read publish one.

What is an llms.txt file for?

llms.txt is a Markdown file at the root of a site, written for AI tools. I counted a site only if its file returned cleanly, wasn’t an HTML page, and opened with a Markdown heading.

Google doesn’t require one. Google says: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features. There’s also no special schema.org structured data that you need to add.”

So the file is optional. The firewall rule isn’t. It decides whether ChatGPT reads anything at all.

Should you block AI training bots and keep llms.txt?

Yes. Blocking training bots is a deliberate choice, and you can leave an existing llms.txt in place. Just don’t spend any more resources on it.

75 of 1,035 llms.txt publishers with a readable robots.txt block at least one AI training bot from the homepage. Training bots collect data for models. Search bots fetch pages to answer questions. OpenAI documents the split: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot.”

The robots.txt version of the welcome-mat problem is rare. 4 of 1,035 llms.txt publishers with a readable robots.txt also tell AI search bots to stay out of the homepage. Treat both robots.txt counts as minimums. I checked the homepage path only.

For the bot-by-bot list, read GPTBot vs OAI-SearchBot: Which AI Crawlers to Allow.

Why do the two files disagree?

This section is opinion. I didn’t survey who owns what at these 34 companies.

Here’s the pattern I’d bet on. Someone on the web or marketing side hears that AI tools want an llms.txt, and publishes one. Someone on the security side turns on bot protection, and AI search requests start bouncing. Each change makes sense on its own. Nobody checks them together.

The robots.txt numbers fit that story. Robots.txt lives with the web team, and only 4 of 1,035 llms.txt publishers block search bots there. The 13 conflicts sit at the firewall.

Does llms.txt work if ChatGPT is blocked?

ChatGPT got a 403 or a bot check on these homepages two days running. A firewall that refuses ChatGPT at the server will likely refuse its request for /llms.txt too. I didn’t test that file request separately.

Even with the door open, the evidence for llms.txt is thin:

  • Ahrefs found 97% of llms.txt files got zero fetches in May 2026.
  • SE Ranking, via Search Engine Journal, looked at about 300,000 domains and found no relationship between llms.txt and citation frequency.

My data measures adoption only. It says nothing about whether llms.txt earns citations. My advice: if you already have an llms.txt, you can leave it. Don’t spend any more time or resources on it, and don’t build a new one.

A blocked homepage isn’t a missing brand, either. On day one, ChatGPT still quoted 22 of the 34 blocked homepages from its search index. That copy can be stale.

How do you line up robots.txt and your firewall?

  1. Ask ChatGPT, in a temporary chat, to open your homepage and quote the headline.
  2. Repeat on a second day.
  3. Read your robots.txt rules for AI search bots and AI training bots, and decide each group on purpose.
  4. Check your firewall and CDN bot rules for the same AI search bots.
  5. Put one owner on both. If you already have an llms.txt, you can leave it on that owner’s list, but don’t put more work into it.

FAQ

Is it a contradiction to block AI training bots and publish llms.txt?

No. 75 of 1,035 llms.txt publishers with a readable robots.txt do it. Training and search are separate jobs.

Does llms.txt increase AI citations?

My data can’t answer that. Ahrefs and SE Ranking found no evidence it does. If you already have one, you can leave it, but don’t spend any more resources on it.

Why does ChatGPT get blocked on a site with llms.txt?

In my test, the block came from the server, not the file. 13 of the 34 sites that blocked ChatGPT both days publish an llms.txt.

Methods and limits

  • Cohort: US B2B software companies with 100+ employees as classified by Apollo (SIC 7372).
  • ChatGPT tests ran on Oct 29 and Oct 30, 2026, on the 132 sites a simulated-bot test flagged.
  • llms.txt and robots.txt were rechecked live on Oct 30, 2026. Robots.txt was parsed with an RFC 9309 parser, homepage path only.
  • I have no citation, prompt or server log data, and no data on team ownership.

Find out if your files and your firewall agree

The welcome-mat problem is easy to miss from the inside. My AI visibility audit tests your site with the real tools and shows you what AI search sees.

Get your AI visibility audit