# What is robots.txt, and how should I set it up?

> Robots.txt is a plain-text file at the root of your website that asks crawlers which URLs they may fetch. It controls crawling, not indexing, and it is not security: a blocked page can still appear in search, and the file is public. For most small businesses a short file that allows everything, blocks only admin areas and lists the sitemap is the safe default.

Topic: Technical SEO. Last reviewed: 12 November 2026. Source: Arcline, https://arcline.celira.space/faq/robots-txt

## What is robots.txt, and what does it do?

Robots.txt is a plain-text file at the root of your website, for example `https://www.example.com/robots.txt`. Before a well-behaved crawler fetches your pages, it reads this file for rules about which addresses it may request. Each rule names a crawler and a path. The file applies only to the host it sits on, so a subdomain such as `shop.example.com` needs its own. Googlebot and Bingbot follow it, and nothing forces a bad bot to.

## What can robots.txt not do?

It controls crawling, not indexing. If you block a page, Google cannot read it, but it can still list the address in results, without a description, when other sites link to it. To keep a page out of search, leave it crawlable and use a [noindex tag](https://arcline.celira.space/faq/noindex-tag), or put it behind a login.

> **Do not use robots.txt to hide private pages.** The file is public, so anyone can read it. A line such as `Disallow: /private-offers/` tells the world exactly where to look. Protect private pages with a login, not with a polite request.

## What does robots.txt syntax look like?

| Line | What it does |
| --- | --- |
| `User-agent: *` | Starts a group of rules for every crawler. Use a name such as `Googlebot` for one crawler. |
| `Disallow: /cart/` | Asks crawlers not to fetch addresses that begin with `/cart/`. |
| `Allow: /wp-admin/admin-ajax.php` | Makes an exception inside a blocked folder. The most specific matching rule wins. |
| `Disallow:` with nothing after it | Blocks nothing. |
| `Disallow: /` | Blocks the entire site. Rarely what you want. |
| `Sitemap: https://www.example.com/sitemap.xml` | Tells crawlers where your sitemap is. Use the full address. |

Paths are case-sensitive and match from the start, so `Disallow: /blog` also blocks `/blog-news/`. Google also understands `*` as a wildcard and `$` as the end of an address.

## What is a safe robots.txt example for a small business?

Allow everything that matters, block only areas with no search value, and point to your [sitemap](https://arcline.celira.space/faq/xml-sitemap). For a WordPress site that can be these four lines, with your own address in the last one:

- `User-agent: *`
- `Disallow: /wp-admin/`
- `Allow: /wp-admin/admin-ajax.php`
- `Sitemap: https://www.example.com/sitemap.xml`

Add `Disallow:` lines for cart, checkout and internal search pages if you have them. Do not block your CSS, JavaScript or image folders, because Google needs them to see pages as visitors do. If you have nothing to block, a file with only the `Sitemap:` line, or no file at all, is fine.

## How do you test robots.txt in Search Console?

1. Open `/robots.txt` in your browser and read it. It should load as plain text, not an error page.
2. In Search Console, open the robots.txt report under Settings. It shows the file Google last fetched, when, and any errors or warnings.
3. Run the URL Inspection tool on an important page. It says whether Google may crawl it or robots.txt blocks it.
4. Re-test after every edit. Google generally caches the file for up to a day, so changes take a little while to show.

## What about AI crawlers in robots.txt?

Robots.txt is also where you allow or block AI crawlers, each with its own name. For example, `User-agent: GPTBot` followed by `Disallow: /` blocks OpenAI's training crawler. The names to know, as of late 2026, are:

- OpenAI: `GPTBot` (training), `OAI-SearchBot` (ChatGPT search), `ChatGPT-User` (fetches a page when a person asks)
- Anthropic: `ClaudeBot`, `Claude-SearchBot`, `Claude-User`. Perplexity: `PerplexityBot`, `Perplexity-User`
- Google: `Google-Extended` (Gemini, no effect on Search ranking). Apple: `Applebot-Extended`. Common Crawl: `CCBot`

Blocking training bots does not remove you from Google Search, and OpenAI says robots.txt rules may not apply to `ChatGPT-User`. Check each vendor's page, because names change. The choice is a business decision: see [should you allow or block GPTBot and other AI crawlers](https://arcline.celira.space/faq/block-or-allow-ai-crawlers).

## What are the classic robots.txt disasters?

- A leftover `Disallow: /` from a staging site that went live with its file. The whole site stops being crawled, and it can go unnoticed for weeks. It is a classic cause of a [website not showing on Google](https://arcline.celira.space/faq/website-not-showing-on-google).
- A CMS setting that discourages search engines, left on after launch.
- Blocking CSS, JavaScript or image folders, so Google cannot render pages properly.
- Using the file to remove pages from Google. It cannot. Use noindex or a removal request.

Arcline reads your robots.txt on every scan and flags rules that block the crawlers that answer people's questions, such as ChatGPT, Perplexity and Claude. Editing robots.txt is a sensitive change, so Arcline prepares the corrected file and applies it only after your approval. See [how the checks work](https://arcline.celira.space/features).

## Related questions

- [Should I allow or block GPTBot and other AI crawlers?](https://arcline.celira.space/faq/block-or-allow-ai-crawlers.md)
- [What is an XML sitemap, and how do I submit it to Google?](https://arcline.celira.space/faq/xml-sitemap.md)
- [What does the noindex tag do, and when should I use it?](https://arcline.celira.space/faq/noindex-tag.md)
- [How do I get my website indexed by Google?](https://arcline.celira.space/faq/get-indexed-on-google.md)
- [Why is my website not showing on Google?](https://arcline.celira.space/faq/website-not-showing-on-google.md)