In my test, the firewall was the problem. Robots.txt wasn’t.
I asked ChatGPT to open 132 B2B software sites that a bot test had flagged. 34 blocked it on both Oct 29 and Oct 30, 2026. Of the 31 with a readable robots.txt, 29 invited AI search bots in.
I call it the bouncer problem. Robots.txt is the sign on the door. The firewall is the bouncer, and the bouncer never read the sign.
This piece goes deeper on one section of my crawl of B2B software websites. Whether you call the work GEO or AEO, access comes first.
How did I test whether ChatGPT can open a site?
I started with a simulated-bot test. It reached 3,263 sites and flagged 132 where an AI bot user agent got refused.
A simulated test proves nothing on its own. So I asked the real tool. In a temporary chat, I asked ChatGPT to open each flagged URL and quote the headline. I ran that on Oct 29, 2026, retested every block on Oct 30, and saved a screenshot of every verdict.
What did ChatGPT see on day one?
On Oct 29, the 132 flagged sites split like this:
- Opened: 79
- Blocked (403 or bot check): 35
- Fetch error: 13
- Unclear: 4
- Inconsistent: 1 (opened Oct 28, blocked Oct 29), excluded
What ChatGPT did on day one with 132 flagged sites
Sites a simulated-bot test flagged, opened by ChatGPT on Oct 29, 2026. Base: 132 flagged sites, not the full cohort.
Source: Scott King crawl of 3,877 B2B software websites. ChatGPT temporary-chat tests, Oct 29, 2026. The inconsistent site opened Oct 28 and was blocked Oct 29, so it is excluded from the blocked count.
79 of 132 opened. The bot test flagged those sites, and the real ChatGPT walked straight in. I come back to that in the last section.
How many stayed blocked on day two?
I retested the 35 day-one blocks on Oct 30. 34 blocked ChatGPT again. One site flipped between days and one returned an unclear “restricted URL” message, and I left both out.
34 flagged sites blocked ChatGPT on both days
Retest on Oct 30, 2026. Base: 132 flagged sites, not a rate for B2B software as a whole.
Source: Scott King crawl of 3,877 B2B software websites. ChatGPT temporary-chat tests, Oct 29 and Oct 30, 2026. Only sites a simulated-bot test flagged were tested.
Keep the base in mind. These 132 sites are the ones a bot test flagged. 34 of 132 tells you what happened to that group. It isn’t a rate for B2B software as a whole.
Why is ChatGPT blocked when robots.txt allows AI crawlers?
Because two different systems answer the request.
Robots.txt is a public file of rules. Bots read it and choose to follow it. A firewall or CDN sits in front of the server and decides, request by request, who gets through. A rule in one doesn’t change the other.
3 of the 34 two-day blocks had no readable robots.txt. Of the other 31, 29 allowed AI search bots on the homepage path. Those sites said yes in writing and no at the server.
The platforms ask for both doors open:
- OpenAI recommends “allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below.”
- Perplexity says “If you’re using a Web Application Firewall (WAF) to protect your site, you may need to explicitly allow Perplexity’s bots.”
- Google asks you to make sure crawling is allowed “in robots.txt, and by any CDN or hosting infrastructure.”
If you’re deciding which bots belong on your allow list, start with GPTBot vs OAI-SearchBot: Which AI Crawlers to Allow.
Does a 403 make your brand invisible in ChatGPT?
No. A 403 means ChatGPT couldn’t open the page at that moment.
ChatGPT keeps a search index. On day one, ChatGPT still quoted 22 of the 34 blocked homepages from its search index.
That cuts both ways. Your brand still shows up. But ChatGPT is quoting an older copy, and that copy can be stale. If you changed your positioning, pricing or product names since the last good fetch, ChatGPT is working from the old version.
This test says nothing about citations. It shows whether ChatGPT can open a page, not how often it recommends you.
Can ChatGPT access my website? How to check
Run the real test. A bot checker can mislead you: real ChatGPT opened 79 of the 132 sites a simulated-bot test had flagged.
- Open a temporary chat in ChatGPT.
- Ask it to open your homepage and quote the headline.
- Repeat on a second day.
- If it fails both times, check your firewall and CDN rules for AI search bots.
- Allow the vendors’ published IP ranges, not only their user agents.
- Run the test again.
Hand step 4 to whoever owns the firewall.
FAQ
Why is ChatGPT getting a 403 on my website?
A firewall, CDN or bot-protection rule refused the request. In my test, 29 of 31 two-day-blocked sites with a readable robots.txt allowed AI search bots there.
Does blocking ChatGPT remove my site from ChatGPT?
My test doesn’t show that. On day one, ChatGPT still quoted 22 of the 34 blocked homepages from its search index. That copy can go stale.
Is robots.txt enough to let ChatGPT in?
No. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.
Methods and limits
- Cohort: US B2B software companies with 100+ employees as classified by Apollo (SIC 7372).
- Only the 132 sites a simulated-bot test flagged were tested in ChatGPT, on Oct 29 and Oct 30, 2026.
- Robots.txt was rechecked live on Oct 30 with an RFC 9309 parser, homepage path only.
- I read the search-index quotes by hand from screenshots.
- I have no citation, prompt or server log data.
Find out if ChatGPT can open your site
The bouncer problem hides in plain sight. My AI visibility audit tests your site with the real tools and shows you what AI search sees.
Scott King helps B2B technology companies build AI marketing systems and create differentiated content.