Is your website actually letting the AI crawlers in?

Check yoursite.com/robots.txt. If it does not name GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, you have not made a decision about AI visibility, you have inherited a default. Those agents do three different jobs, and blocking the wrong one has effects most people do not intend: you can be excluded from AI answers while still ranking perfectly in Google, and never see it in any report you currently run.

The three jobs, which is where the confusion starts

Nearly every mistake here comes from treating "AI crawler" as one thing. There are three kinds, and they have different consequences.

Training crawlers collect pages to train future models. OpenAI documents its agents and so does every other major operator, which is what makes this checkable rather than guesswork. GPTBot, ClaudeBot, anthropic-ai and CCBot are in this group. Blocking them keeps your content out of the next model. It does nothing to stop an assistant answering a question about you today.

Search indexers build the index an assistant searches when it needs current information. OAI-SearchBot and PerplexityBot are here. This is the group that decides whether you get cited, and it is the one most often blocked by accident.

Live fetchers retrieve a page in the moment because a user asked about it. ChatGPT-User, Claude-User and Perplexity-User do this. Block these and an assistant literally cannot open your page when a buyer asks it to.

So "we blocked the AI bots to protect our content" usually means one of two very different things, and the person who wrote the line often did not know which.

Google-Extended is not a crawler, and this catches people

Per Google's own crawler documentation, Google-Extended does not fetch anything. It is a switch that controls whether what Googlebot already fetched can be used by Gemini and AI Overviews.

That means disallowing Google-Extended keeps you in Google search and takes you out of Google's AI answers. For most businesses that is exactly backwards, and because your search rankings do not move, nothing in your reporting tells you it happened.

If you only change one line after reading this, make it that one.

The Allow-list trap, using our own file as the example

Here is the mistake we found on our own site while writing this, which is a better example than a hypothetical one.

nbh.co/robots.txt names the AI agents properly and gives each a group like this:

User-agent: GPTBot
Allow: /services/
Allow: /work/
Allow: /learn/
Allow: /about/

That reads like a restriction. It is not one. In the robots exclusion standard, anything not matched by a Disallow is permitted, so a group made only of Allow lines permits the entire site. Those five lines are doing no work at all. GPTBot may crawl everything except the four preview paths we disallow further down.

In our case the outcome is what we wanted anyway, so nothing is broken. But the file expresses an intention it does not enforce, and if we had meant to confine crawlers to four sections, we would have been wrong and had no way of knowing.

The rule worth remembering: Allow does not restrict, it makes an exception inside a Disallow. If you want to confine a crawler you need Disallow: / followed by Allow: lines for what you permit. If you want to grant access, the honest version is one Allow: /.

There is a second thing our live file shows. Naming a crawler creates a group for it, and **once a group exists for that agent, the User-agent: * rules no longer apply to it.** So every restriction you thought applied site-wide silently stops applying to every bot you name. Check that the named groups repeat anything that actually matters.

Blocking on purpose is fine. Blocking by accident is not

None of this is an argument for allowing everything. We deliberately block Bytespider, because it is widely reported to ignore crawl-delay and hammer small sites, and no answer engine we care about is fed by it. That is a decision with a reason attached.

The question worth asking is not "should we let AI use our content", it is "which of these three jobs do we want to be part of". Most Australian businesses we look at want to be found and cited, are relaxed about search indexing, and have opinions about training. That is a perfectly coherent position and it is expressible in about fifteen lines.

The trap is arriving at a position by inheritance. A robots.txt written in 2019 makes a 2026 decision by omission.

While you are in there, check llms.txt actually serves

We found this on our own site too. nbh.co/llms.txt returns a 301 to a HubSpot file manager URL on a different hostname. The file exists and is served, but a crawler asking for the file at the site root gets a cross-domain redirect, and not every client follows those for well-known files.

If you have written an llms.txt, curl it and read the status code rather than trusting the browser, which follows the redirect happily and shows you a working file.

What we do on client builds

• Name every agent explicitly. A bare User-agent: * / Allow: / reads as an absence of a decision. Naming each one puts the choice on the record so the next person can see what was considered.
• Write the reason in a comment. Our staging file carries a comment block explaining which agents were allowed and why, including why Bytespider is blocked. It has already stopped one person "tidying" the block into the allow list.
• Check the named groups, not just the star group. This is the one that bites, because the star group looks comprehensive and is being ignored for every bot you name.
• Test with the status code. Fetch robots.txt and llms.txt with curl and read what comes back, because the browser hides redirects.

The short version

Open your robots.txt now. If GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot are not named, you have a default rather than a decision. Check Google-Extended first, because disallowing it removes you from Gemini and AI Overviews while leaving your Google rankings untouched, which makes it invisible in every report you run.

Then remember that Allow on its own restricts nothing, and that naming an agent exempts it from your User-agent: * rules. Both of those are in our own live file, which is how we know to look.

Frequently asked questions

Will blocking GPTBot stop ChatGPT talking about my business? No. GPTBot is a training crawler. Blocking it keeps your pages out of future model training and has no effect on an assistant answering a question about you today, which is handled by the search indexer and the live fetcher.

Does robots.txt actually stop these crawlers? It is a voluntary standard. The major operators, OpenAI, Anthropic, Google and Perplexity, publish their agent names and honour it. Some crawlers are widely reported not to. If you need enforcement rather than a request, that is a firewall or WAF rule, not a robots.txt line.

Should I block AI crawlers to protect my content? It depends which job you mean. Blocking training crawlers is a reasonable position if your content is your product. Blocking search indexers means giving up being cited, which for most businesses is the whole point of publishing. Decide the two separately.

How do I know if any of this is working? Server logs are the direct answer, since they show which agents actually fetched what. Beyond that, tracking whether assistants name you for your key questions is the practical measure, which we covered in measuring AI search visibility.

Sources

• OpenAI, GPTBot and OAI-SearchBot documentation. The distinction between training, search indexing and live browsing agents.
• Anthropic, ClaudeBot documentation. Current and legacy agent names.
• Perplexity, PerplexityBot documentation.
• Google, Google-Extended. Confirms it is a usage control over content Googlebot has already fetched, not a separate crawler.
• nbh.co/robots.txt and v2.nbh.co/robots.txt, both read 1 September 2026. The Allow-list behaviour and the llms.txt redirect described here were found in our own files.

We found two things wrong with our own setup while writing this, which is roughly the hit rate we expect. If you want a second pair of eyes on yours, give us a shout and tell us what's broken.