The path to being cited by AI search
Start with the steps in order: let the right crawlers in, make your pages easy to quote, then build it into your site. Every guide opens with the short answer. Updated .
Let AI crawlers in
Which bots read your site, which ones decide whether you get cited, and the robots.txt rules for each.
AI crawlers list: every bot name for robots.txt
3 min readTo be cited in AI answers, allow the search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity) and Googlebot (AI Overviews and AI Mode). GPTBot, ClaudeBot, Google-Extended and CCBot are about model training, and blocking them doesn't remove you from AI search.
Start hereBlock AI training but stay in AI search answers
2 min readDisallow GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot in robots.txt, and keep OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot and Googlebot allowed. The full file is below.
Does blocking AI crawlers hurt your Google rankings?
2 min readNo. Blocking AI training crawlers, including Google's own Google-Extended token, doesn't change your Google rankings, and it doesn't remove you from AI Overviews or AI Mode. Only blocking Googlebot does that, and that removes you from Google Search altogether.
How to allow GPTBot (and ChatGPT search) in robots.txt
2 min readAdd a User-agent: GPTBot group with Allow: / to robots.txt, and make sure no broader rule blocks it. If you want to appear in ChatGPT's search answers, allow OAI-SearchBot too. It's a separate crawler, and it's the one that decides whether ChatGPT search can show your pages.
Make pages easy to quote
What assistants can actually read, and how to write pages they can lift a clear answer from.
Can AI crawlers see JavaScript content?
2 min readMostly no. A 2024 study of crawler traffic by Vercel and MERJ found that OpenAI's crawlers, ClaudeBot, PerplexityBot, Meta's crawler, Bytespider and CCBot download JavaScript files but don't execute them. Google's crawlers, which Gemini also uses, and Applebot do render JavaScript.
What is llms.txt, and do you need one?
2 min readLlms.txt is an optional Markdown file at the root of your site that summarizes what the site is and links to its most useful pages. Nothing you can measure today depends on it. It's cheap to add, so add a short one and move on.
Writing pages AI assistants can quote
2 min readPut a direct answer to the page's main question in the first few sentences, give each section a heading that says what it answers, and back claims with specifics you can stand behind. That's ordinary good writing, and it's what assistants quote.
Build it into your site
The structured data, rendering and files that make all of the above automatic.
Check where your site stands
Run the free check on any page to see which of these steps it already passes.