Let AI crawlers in
Which bots read your site, which ones decide whether you get cited, and the robots.txt rules for each.
4 guides, last updated
AI crawlers list: every bot name for robots.txt
3 min readTo be cited in AI answers, allow the search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and Googlebot (AI Overviews and AI Mode). GPTBot, ClaudeBot, Google-Extended and CCBot are about model training, and blocking them doesn't remove you from AI search.
Block AI training but stay in AI search answers
2 min readDisallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot in robots.txt, and keep OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot allowed. The full file is below.
Does blocking AI crawlers hurt your Google rankings?
2 min readNo. Blocking AI training crawlers, including Google's own Google-Extended token, doesn't change your Google rankings, and it doesn't remove you from AI Overviews or AI Mode. Only blocking Googlebot does that, and that removes you from Google Search altogether.
How to allow GPTBot (and ChatGPT search) in robots.txt
2 min readAdd a User-agent: GPTBot group with Allow: / to robots.txt, and make sure no broader rule blocks it. If you want to appear in ChatGPT's search answers, allow OAI-SearchBot too. It's a separate crawler, and it's the one that decides whether ChatGPT search can show your pages.