Short answer
For most businesses that want customers to find them, allow the crawlers behind AI search, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, and treat blocking training crawlers such as GPTBot as a separate, optional choice. Blocking GPTBot does not remove you from Google, and Google-Extended does not affect Search ranking. Decide bot by bot, not all at once.
Which AI crawlers exist, and what is each for?
Each AI company runs several crawlers with different jobs, and you control each by its own name, its robots.txt token. Check each vendor's own page before relying on this list, because names and behavior change.
| Crawler | Company | What it is for |
|---|---|---|
GPTBot | OpenAI | Collecting content that may be used to train models |
OAI-SearchBot | OpenAI | Building the index behind ChatGPT search |
ChatGPT-User | OpenAI | Fetching a page when a person asks ChatGPT to look at it |
ClaudeBot | Anthropic | Collecting content for training |
Claude-SearchBot | Anthropic | Search results |
Claude-User | Anthropic | Fetching a page for a person who asked |
PerplexityBot | Perplexity | Building the index Perplexity answers draw on |
Perplexity-User | Perplexity | Fetching a page when a question needs it |
Googlebot | Crawling for Search; AI Overviews and AI Mode run on Search ranking systems | |
Google-Extended | A control for Gemini training and grounding; no effect on Search ranking | |
Applebot-Extended | Apple | A control for whether Apple may train its models on your content |
CCBot | Common Crawl | An open web archive that many AI training sets draw on |
Training, search and user-triggered: why the difference matters
- Training crawlers (GPTBot, ClaudeBot, CCBot) gather text to train models. Blocking them stops future collection by those bots. It does not erase content already collected.
- Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build the indexes assistants search when they answer. Block one and that assistant has less reason to show you.
- User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) load a page when a person asks about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person made the request.
That split is why “block all AI bots” is a blunt tool. It can remove you from the answers where customers ask about you while doing little about training that has already happened. How assistants then pick sources is covered in how AI assistants decide which websites to cite.
What does blocking do, and what does it not do?
- It does ask well-behaved crawlers not to fetch your pages. It can keep you out of a company's future training data, or out of its search answers if you block its search bot.
- It does not remove you from Google. Blocking GPTBot has no effect on Google, and Google-Extended controls Gemini use only, not Search ranking.
- It does not opt you out of AI Overviews or AI Mode. Google says there is no separate opt-out beyond existing Search controls (robots.txt blocking for Googlebot, noindex and snippet controls), which also change how you appear in ordinary Search.
- It does not delete what a company already collected, or stop a crawler that ignores the rules. robots.txt is a request, not a lock; only a login or firewall rule enforces access.
Which choice fits my kind of business?
| Business | Search and user bots | Training bots | Reasoning |
|---|---|---|---|
| Local business or clinic | Allow | Allow, or your choice | Customers ask assistants for places and services, so being named is the goal. |
| Online store or SaaS | Allow | Allow, or block by policy | Mentions and citations bring discovery; training use is a policy call. |
| Publisher or ad-funded site | Allow, review often | Often block | Answers can cut visits, so some publishers block training and seek licensing. |
| Paywalled or course content | Public pages only | Block | Paid material belongs behind a login, not only a robots.txt rule. |
| Photographer, author or researcher | Allow | Block if you object | Your original work is your product. |
| Staging or private areas | Block | Block | Use a login; robots.txt is not security. |
This is general guidance, not legal advice. If copyright or licensing is central to your business, take advice before choosing.
What does a sample robots.txt look like?
- Open your robots.txt
It lives at the root, for example
yourdomain.com/robots.txt. Keep your existingUser-agent: *rules. - Block training only
Add a group of two lines:
User-agent: GPTBotthenDisallow: /. Repeat the pair forClaudeBot,CCBot,Google-ExtendedandApplebot-Extendedif you want to opt out of all of them. - Keep answer engines in
Never list
OAI-SearchBot,PerplexityBotorClaude-SearchBotunder a Disallow rule. To make it explicit, give each its own group withAllow: /. - Mind the grouping
A crawler follows only the most specific group that names it, so a bot with its own group ignores the
*group. End the file with aSitemap:line giving your sitemap address. - Test it
Open the file in a browser, confirm your host's bot protection is not blocking the same crawlers, and re-check after a few days.
What should I do next?
Start by checking what happens today: open your robots.txt and your host's bot settings. Arcline's scan flags rules that block the answer crawlers as urgent and prepares a corrected file for your approval, and treats admitting training crawlers as your own choice, flagged separately; see the docs. For the basics of the file, read what robots.txt is and how to set it up; the separate llms.txt file does not control crawlers.