# Should I allow or block GPTBot and other AI crawlers?

> For most businesses that want customers to find them, allow the crawlers behind AI search, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, and treat blocking training crawlers such as GPTBot as a separate, optional choice. Blocking GPTBot does not remove you from Google, and Google-Extended does not affect Search ranking. Decide bot by bot, not all at once.

Topic: AI search: AEO and GEO. Last reviewed: 12 November 2026. Source: Arcline, https://arcline.celira.space/faq/block-or-allow-ai-crawlers

## Which AI crawlers exist, and what is each for?

Each AI company runs several crawlers with different jobs, and you control each by its own name, its robots.txt token. Check each vendor's own page before relying on this list, because names and behavior change.

| Crawler | Company | What it is for |
| --- | --- | --- |
| `GPTBot` | OpenAI | Collecting content that may be used to train models |
| `OAI-SearchBot` | OpenAI | Building the index behind ChatGPT search |
| `ChatGPT-User` | OpenAI | Fetching a page when a person asks ChatGPT to look at it |
| `ClaudeBot` | Anthropic | Collecting content for training |
| `Claude-SearchBot` | Anthropic | Search results |
| `Claude-User` | Anthropic | Fetching a page for a person who asked |
| `PerplexityBot` | Perplexity | Building the index Perplexity answers draw on |
| `Perplexity-User` | Perplexity | Fetching a page when a question needs it |
| `Googlebot` | Google | Crawling for Search; AI Overviews and AI Mode run on Search ranking systems |
| `Google-Extended` | Google | A control for Gemini training and grounding; no effect on Search ranking |
| `Applebot-Extended` | Apple | A control for whether Apple may train its models on your content |
| `CCBot` | Common Crawl | An open web archive that many AI training sets draw on |

## Training, search and user-triggered: why the difference matters

- **Training crawlers** (GPTBot, ClaudeBot, CCBot) gather text to train models. Blocking them stops future collection by those bots. It does not erase content already collected.
- **Search crawlers** (OAI-SearchBot, PerplexityBot, Claude-SearchBot) build the indexes assistants search when they answer. Block one and that assistant has less reason to show you.
- **User-triggered fetchers** (ChatGPT-User, Perplexity-User, Claude-User) load a page when a person asks about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person made the request.

That split is why “block all AI bots” is a blunt tool. It can remove you from the answers where customers ask about you while doing little about training that has already happened. How assistants then pick sources is covered in [how AI assistants decide which websites to cite](https://arcline.celira.space/faq/how-ai-chooses-sources).

## What does blocking do, and what does it not do?

- **It does** ask well-behaved crawlers not to fetch your pages. It can keep you out of a company's future training data, or out of its search answers if you block its search bot.
- **It does not** remove you from Google. Blocking GPTBot has no effect on Google, and Google-Extended controls Gemini use only, not Search ranking.
- **It does not** opt you out of AI Overviews or AI Mode. Google says there is no separate opt-out beyond existing Search controls (robots.txt blocking for Googlebot, noindex and snippet controls), which also change how you appear in ordinary Search.
- **It does not** delete what a company already collected, or stop a crawler that ignores the rules. robots.txt is a request, not a lock; only a login or firewall rule enforces access.

> Your host can override a friendly robots.txt. Bot protection, such as a firewall or a Cloudflare AI crawler setting, can turn assistants away before they read the file, so test both layers.

## Which choice fits my kind of business?

| Business | Search and user bots | Training bots | Reasoning |
| --- | --- | --- | --- |
| Local business or clinic | Allow | Allow, or your choice | Customers ask assistants for places and services, so being named is the goal. |
| Online store or SaaS | Allow | Allow, or block by policy | Mentions and citations bring discovery; training use is a policy call. |
| Publisher or ad-funded site | Allow, review often | Often block | Answers can cut visits, so some publishers block training and seek licensing. |
| Paywalled or course content | Public pages only | Block | Paid material belongs behind a login, not only a robots.txt rule. |
| Photographer, author or researcher | Allow | Block if you object | Your original work is your product. |
| Staging or private areas | Block | Block | Use a login; robots.txt is not security. |

This is general guidance, not legal advice. If copyright or licensing is central to your business, take advice before choosing.

## What does a sample robots.txt look like?

1. **Open your robots.txt.** It lives at the root, for example `yourdomain.com/robots.txt`. Keep your existing `User-agent: *` rules.
2. **Block training only.** Add a group of two lines: `User-agent: GPTBot` then `Disallow: /`. Repeat the pair for `ClaudeBot`, `CCBot`, `Google-Extended` and `Applebot-Extended` if you want to opt out of all of them.
3. **Keep answer engines in.** Never list `OAI-SearchBot`, `PerplexityBot` or `Claude-SearchBot` under a Disallow rule. To make it explicit, give each its own group with `Allow: /`.
4. **Mind the grouping.** A crawler follows only the most specific group that names it, so a bot with its own group ignores the `*` group. End the file with a `Sitemap:` line giving your sitemap address.
5. **Test it.** Open the file in a browser, confirm your host's bot protection is not blocking the same crawlers, and re-check after a few days.

## What should I do next?

Start by checking what happens today: open your robots.txt and your host's bot settings. Arcline's scan flags rules that block the answer crawlers as urgent and prepares a corrected file for your approval, and treats admitting training crawlers as your own choice, flagged separately; see [the docs](https://arcline.celira.space/docs). For the basics of the file, read [what robots.txt is and how to set it up](https://arcline.celira.space/faq/robots-txt); the separate [llms.txt](https://arcline.celira.space/faq/does-llms-txt-work) file does not control crawlers.

## Related questions

- [What is robots.txt, and how should I set it up?](https://arcline.celira.space/faq/robots-txt.md)
- [Does llms.txt work, and do I need one?](https://arcline.celira.space/faq/does-llms-txt-work.md)
- [How do I optimize my website for Perplexity and other AI search engines?](https://arcline.celira.space/faq/perplexity-seo.md)
- [How do I get my website to show up in ChatGPT answers?](https://arcline.celira.space/faq/rank-in-chatgpt.md)
- [How do AI assistants decide which websites to cite?](https://arcline.celira.space/faq/how-ai-chooses-sources.md)