Not all AI bots are the same
The phrase “AI bot” covers three different jobs, and each needs its own decision. Some crawlers index your site for AI search, others collect material to train models, and others open a page because a person has just asked an assistant to read it.
The big companies split these crawlers into separate User-agent strings, so you can manage them by purpose. Bot lists change, so check the primary sources before editing the file.
| Purpose | Bots | What it means for your site |
|---|---|---|
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, meta-webindexer | Your site appears in AI search answers and links |
| Training | GPTBot, ClaudeBot, meta-externalagent | Your material may be used to train models |
| On user request | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher | The page is opened because a person asked for it |
| Google token | Google-Extended | A robots.txt token with no User-agent string of its own: controls use for Gemini, does not affect Google Search |
Two separate switches
The question “should I let AI in” is really two: do you want mentions and visits from AI search, and are you fine with your text going into model training. The decisions are independent: you can let search bots in and block training bots at the same time.
A ready-made robots.txt
The sample below allows search bots and blocks training bots. For a different goal, move lines between the groups. The file lives in the site root at /robots.txt. A crawler looks for the group with its own name and, if it finds one, the other groups do not apply to it.
# Search: show the site in AI answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
Disallow: /admin/
# Training: do not use the site material
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: Google-Extended
Disallow: /
# Everyone else
User-agent: *
Disallow: /admin/The Disallow: /admin/ line is repeated in the search group on purpose: rules of the wildcard group do not apply to a bot that has a group of its own.
OpenAI says robots.txt changes take effect after about 24 hours, so do not check the result straight away.
Bots that come at a person’s request
ChatGPT-User, Claude-User, Perplexity-User and meta-externalfetcher do not follow a schedule; they arrive when a person asks an assistant to open a specific link. OpenAI states plainly that in this case robots.txt rules may not apply, and Meta says meta-externalfetcher may bypass them.
The practical conclusion: robots.txt is a request, not a lock. If a page must not be opened by outsiders, protect it with authentication or a server-level rule. Writing such rules is covered in the article on Cloudflare WAF custom rules.
Cloudflare: settings and pay per crawl
If your site sits behind Cloudflare, some decisions can be made in the dashboard without editing files. The path as of November 2026 is Security → Settings → AI Crawler and Agent access policies. The section used to be called “Configure AI bot policies”; the rename is recorded in the Cloudflare changelog of 4 November 2026. Names keep changing, so look for the section by meaning.
Pay per crawl is a mechanism where the publisher sets a price per request and picks Allow, Charge or Block for each crawler. For a paid page the crawler receives an HTTP 402 response, and requests are signed with the Web Bot Auth scheme. The status today is a private beta, so it is too early to build a revenue plan on it.
Cloudflare has changed its defaults twice. On 1 July 2025 the company announced blocking AI crawlers by default unless they pay. On 1 July 2026 it announced that from 15 September 2026 it would split AI traffic into three classes: Search, Agent and Training. How this applies to your particular domain is not visible from the documentation, so check your current settings yourself. A general overview of the section is in the article on bot management in Cloudflare.
Blocking by IP address
Blocking bots with a list of IP addresses is unreliable, and Anthropic warns about this directly: in its words, such blocking may not work correctly and does not guarantee a persistent opt-out. A WAF rule on the User-agent string is not absolute either: the string is easy to fake.
llms.txt: what is known and what is not
llms.txt is an informal proposal by Jeremy Howard from llmstxt.org: put a Markdown overview file for language models in the site root. It is not an IETF or W3C standard. We could not find confirmation that even one major search engine or model developer reads it.
Google states in its guide to optimizing for generative search that Google Search does not use such files and that creating one will neither harm nor help visibility in Google. That applies to Google only. For the other systems it is more honest to say “no confirmation” than “does not work”: third-party articles claim the opposite in both directions, but they have no primary sources.
Bottom line: the file is cheap, but do not bet your plans on it. Start with robots.txt and the quality of the pages themselves; for the threats AI agents pose on your side, read the article on AI agent security and prompt injection.
Where to start
Do not change the file blindly: first see who actually visits you, and only then decide.
- Decide what matters more: mentions in AI search or protecting your text from training.
- Count the requests in the access log.
- Add the groups you need to
robots.txt. - After a day, repeat the count and compare.
Frequently asked questions
If I block GPTBot, will I disappear from ChatGPT search?
No. According to OpenAI’s documentation, GPTBot belongs to model training, while showing sites in ChatGPT search is handled by a separate OAI-SearchBot. These are two different switches.
Will I drop out of Google if I block Google-Extended?
No. Google says this token does not affect whether a site is included in Google Search. The documentation also does not say that it controls AI overviews; to limit display in Search, Google names nosnippet, max-snippet and noindex.
What should ZevsHost customers do?
The robots.txt file sits in the site root, and on shared hosting you edit it in the panel file manager: no server settings are needed. On a VPS or dedicated server you configure WAF rules, logs and web-server-level protection yourself.