# Surfacepath > Free AI readiness check. See whether ChatGPT, Claude, Perplexity and Google AI answers can reach, read and cite your pages, with the exact fixes. Surfacepath checks a URL the way AI crawlers do: it reads robots.txt for each AI crawler, fetches the page without running JavaScript, and checks indexing rules, structured data, headings, dates, the sitemap and llms.txt. ## Tool - [AI readiness check](https://surfacepath.com/): enter a URL, get a score out of 100 and fixes - [How the check works](https://surfacepath.com/methodology): what is fetched and how each item is scored ## AI crawlers - [AI crawlers list: every bot name for robots.txt](https://surfacepath.com/guides/ai-crawlers-list): The user agents OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl use, what each one does, and which ones decide whether AI answers can cite you. - [Block AI training but stay in AI search answers](https://surfacepath.com/guides/block-ai-training-keep-ai-search): A robots.txt that opts out of model training while keeping your pages available to ChatGPT search, Claude, Perplexity and Google AI Overviews. - [Does blocking AI crawlers hurt your Google rankings?](https://surfacepath.com/guides/google-extended-ai-overviews): What Google-Extended does and doesn't control, how AI Overviews choose pages, and the settings that actually limit what Google shows. - [How to allow GPTBot (and ChatGPT search) in robots.txt](https://surfacepath.com/guides/allow-gptbot-robots-txt): The exact robots.txt lines to let OpenAI's crawlers in, the difference between GPTBot and OAI-SearchBot, and how to set it up in Next.js. ## AI readiness - [Can AI crawlers see JavaScript content?](https://surfacepath.com/guides/javascript-ai-crawlers): Most AI crawlers don't run JavaScript. What that means for React and Next.js sites, and how to make sure your content is in the HTML. - [What is llms.txt, and do you need one?](https://surfacepath.com/guides/llms-txt): What the llms.txt proposal is, what Google has said about it, and why it is worth ten minutes but not more. - [Writing pages AI assistants can quote](https://surfacepath.com/guides/answer-first-content): How to structure a page so search engines and AI assistants can lift a clear answer from it, without writing for robots. ## Next.js - [Structured data for AI answers in Next.js](https://surfacepath.com/guides/structured-data-nextjs): Which schema.org types are worth adding, what they do and don't do for AI search, and how to add JSON-LD in the Next.js App Router. ## Glossary - [AEO (answer engine optimization)](https://surfacepath.com/glossary/aeo): AEO, or answer engine optimization, is the work of making your content easy for AI assistants and answer features to find, understand and quote, so you appear in answers from ChatGPT, Claude, Perplexity and Google's AI Overviews. - [AI Overviews](https://surfacepath.com/glossary/ai-overviews): AI Overviews are the AI-written summaries Google shows at the top of some search results, with links to the pages they draw on. They are built from Google's normal search index. - [Canonical URL](https://surfacepath.com/glossary/canonical-url): A canonical URL is the preferred address of a page when the same content is reachable at several addresses. You declare it with a link rel="canonical" tag so search engines combine signals on one URL. - [Claude-SearchBot](https://surfacepath.com/glossary/claude-searchbot): Claude-SearchBot is Anthropic's crawler that indexes web pages to improve Claude's search results. Blocking it can reduce how often your content appears when Claude searches the web. - [ClaudeBot](https://surfacepath.com/glossary/claudebot): ClaudeBot is Anthropic's crawler for collecting public web content that may be used to train its models. Blocking it excludes your future content from training; Claude's search uses a separate crawler, Claude-SearchBot. - [GEO (generative engine optimization)](https://surfacepath.com/glossary/geo): GEO, or generative engine optimization, is another name for optimizing content to appear in answers written by generative AI, such as ChatGPT, Perplexity and Google's AI Overviews. It means much the same as AEO. - [Google-Extended](https://surfacepath.com/glossary/google-extended): Google-Extended is a robots.txt token that controls whether Google may use your content to train its Gemini models. It is not a separate crawler, and blocking it does not affect Google Search or AI Overviews. - [GPTBot](https://surfacepath.com/glossary/gptbot): GPTBot is OpenAI's crawler for collecting content that may be used to train its models. Blocking it in robots.txt opts your site out of that training; it does not affect ChatGPT search, which uses OAI-SearchBot. - [llms.txt](https://surfacepath.com/glossary/llms-txt): llms.txt is an optional Markdown file at the root of a website that summarizes what the site is and links to its most useful pages, proposed in 2024 as a clean map for language models. - [noindex](https://surfacepath.com/glossary/noindex): noindex is a rule, set in a robots meta tag or an X-Robots-Tag header, that tells search engines not to include a page in their results. Pages with noindex are also left out of AI answers built from search indexes. - [nosnippet](https://surfacepath.com/glossary/nosnippet): nosnippet is a robots meta rule that stops search engines from showing any text from a page in results, including in Google's AI Overviews. The page can still rank. - [OAI-SearchBot](https://surfacepath.com/glossary/oai-searchbot): OAI-SearchBot is OpenAI's crawler for ChatGPT search. If robots.txt blocks it, your pages won't appear in ChatGPT's search answers, though they can still show as plain navigational links. - [PerplexityBot](https://surfacepath.com/glossary/perplexitybot): PerplexityBot is Perplexity's crawler for finding and linking websites in its search answers. Perplexity says it is not used to train AI models, and it follows robots.txt. - [robots.txt](https://surfacepath.com/glossary/robots-txt): robots.txt is a plain-text file at the root of a website that tells crawlers which paths they may and may not fetch, using groups of rules per user agent. - [Structured data](https://surfacepath.com/glossary/structured-data): Structured data is machine-readable information about a page, usually written as JSON-LD using schema.org types, that tells search engines and assistants what the page is: an article, a product, an organization and so on. - [User agent](https://surfacepath.com/glossary/user-agent): A user agent is the name a browser or crawler sends to identify itself when it requests a page, such as Googlebot or GPTBot. robots.txt rules are written per user agent. ## Updates - [Google's spam policies now cover AI Overviews and AI Mode](https://surfacepath.com/updates/google-spam-policies-ai-answers) (2026-10-30): Google added 'manipulating generative AI responses' to its definition of spam. What changed, what counts, and what to do if you work on AI visibility. - [What the August 2026 spam update hit](https://surfacepath.com/updates/august-2026-spam-update) (2026-10-30): Case studies from Google's August 2026 spam update point to the same pattern: scaled pages with little value, whether written by people, templates or AI.