Glossary

Short, plain definitions of the words behind AI search and SEO, from crawler names to the rules that decide what assistants can quote.

Filter terms

A

AEO (answer engine optimization): AEO, or answer engine optimization, is the work of making your content easy for AI assistants and answer features to find, understand and quote, so you appear in answers from ChatGPT, Claude, Perplexity and Google's AI Overviews.
AI Overviews: AI Overviews are the AI-written summaries Google shows at the top of some search results, with links to the pages they draw on. They are built from Google's normal search index.

Canonical URL: A canonical URL is the preferred address of a page when the same content is reachable at several addresses. You declare it with a link rel="canonical" tag so search engines combine signals on one URL.
Claude-SearchBot: Claude-SearchBot is Anthropic's crawler that indexes web pages to improve Claude's search results. Blocking it can reduce how often your content appears when Claude searches the web.
ClaudeBot: ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train its models. Blocking it excludes your future content from training; Claude's search uses a separate crawler, Claude-SearchBot.

GEO (generative engine optimization): GEO, or generative engine optimization, is another name for optimizing content to appear in answers written by generative AI, such as ChatGPT, Perplexity and Google's AI Overviews. It means much the same as AEO.
Google-Extended: Google-Extended is a robots.txt token that controls whether Google may use your content to train its Gemini models. It is not a separate crawler, and blocking it does not affect Google Search or AI Overviews.
GPTBot: GPTBot is OpenAI's crawler for collecting content that may be used to train its models. Blocking it in robots.txt opts your site out of that training; it does not affect ChatGPT search, which uses OAI-SearchBot.

llms.txt: llms.txt is an optional Markdown file at the root of a website that summarizes what the site is and links to its most useful pages, proposed in 2024 as a clean map for language models.

noindex: noindex is a rule, set in a robots meta tag or an X-Robots-Tag header, that tells search engines not to include a page in their results. Pages with noindex are also left out of AI answers built from search indexes.
nosnippet: nosnippet is a robots meta rule that stops search engines from showing any text from a page in results, including in Google's AI Overviews. The page can still rank.

OAI-SearchBot: OAI-SearchBot is OpenAI's crawler for ChatGPT search. If robots.txt blocks it, your pages won't appear in ChatGPT's search answers, though they can still show as plain navigational links.

PerplexityBot: PerplexityBot is Perplexity's crawler for finding and linking websites in its search answers. Perplexity says it is not used to train AI models, and it follows robots.txt.

robots.txt: robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may and may not fetch, using groups of rules per user agent.

Structured data: Structured data is machine-readable information about a page, usually written as JSON-LD using schema.org types, that tells search engines and assistants what the page is: an article, a product, an organization and so on.

User agent: A user agent is the name a browser or crawler sends to identify itself when it requests a page, such as Googlebot or GPTBot. robots.txt rules are written per user agent.