To be cited in AI answers, allow the search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and Googlebot (AI Overviews and AI Mode). GPTBot, ClaudeBot, Google-Extended and CCBot are about model training, and blocking them doesn't remove you from AI search.
Most AI companies now run more than one crawler, and each has its own name in robots.txt. They fall into three groups, and only one of them decides whether an assistant can find and cite your pages in search-style answers.
- Search crawlers index pages so an assistant can cite them when it searches the web.
- Training crawlers collect content that may be used to train future models.
- User fetchers open a page because a person asked the assistant to read it.
The crawlers, by company#
| Robots.txt name | Company | What it does | Follows robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search | Yes |
| GPTBot | OpenAI | Collects data for model training | Yes |
| ChatGPT-User | OpenAI | Opens a page a user asks about | May not |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's search results | Yes |
| ClaudeBot | Anthropic | Collects data for model training | Yes |
| Claude-User | Anthropic | Fetches a page a user's question needs | Yes |
| PerplexityBot | Perplexity | Finds and links pages in Perplexity answers | Yes |
| Perplexity-User | Perplexity | Opens a page a user asks about | Generally no |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | |
| Google-Extended | A token that controls use for Gemini training; not a separate crawler | Yes | |
| Applebot | Apple | Search features such as Siri and Spotlight | Yes |
| Applebot-Extended | Apple | A token that controls use for Apple's model training | Yes |
| CCBot | Common Crawl | An open web archive that many models are trained on | Yes |
The OpenAI, Anthropic and Perplexity entries come from each company's own crawler documentation, linked at the end of this page. Names are case-insensitive in robots.txt, but copy them exactly to be safe.
Which ones matter for being cited?#
If your goal is to show up in AI answers, the search crawlers are the ones you can't block: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. Blocking GPTBot, ClaudeBot, Google-Extended or CCBot is a separate decision about training. It doesn't remove you from AI search, and our guide on blocking training while staying in AI search has a ready-made robots.txt.
Robots.txt isn't the only gate#
A firewall or CDN can block crawlers before they ever read robots.txt. Cloudflare and other providers offer one-click AI bot blocking, and it is easy to switch on without noticing that it also blocks the search crawlers you want. If our checker gets an HTTP 403 from your site, look at your firewall's bot settings before anything else.
How to tell a real crawler from a fake one#
Anyone can send a request that claims to be GPTBot. OpenAI and Anthropic both publish the IP ranges their crawlers use, so you can confirm that a visitor in your logs is genuine. Anthropic notes that blocking its IP addresses is not a reliable way to opt out, because the crawler can then no longer read your robots.txt.
Sources: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers.
Questions
Which AI crawlers should I allow if I want to be cited?
The search crawlers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity and Googlebot for Google's AI Overviews and AI Mode. Training crawlers such as GPTBot and ClaudeBot don't decide whether you're cited in search answers.
Does robots.txt stop ChatGPT opening my page when a user pastes the link?
Not reliably. OpenAI says ChatGPT-User, which fetches pages a person asks about, may not follow robots.txt because the request is user-initiated. Perplexity says the same of Perplexity-User.
Is blocking by IP address a good alternative?
Anthropic advises against it: if you block its IP addresses, its crawler can't read your robots.txt at all. Use robots.txt for rules and the operators' published IP lists only to verify that a visitor is genuine.