Reading robots.txt, fetching the page without JavaScript, checking structured data…
An example report for astro.build, re-checked daily. Enter your own site above to see how it compares, or compare Astro with other frameworks.
One problem stops assistants from reading or quoting this page. Fix it first.
718 words in the HTML, loaded in 0.1 s.
| Crawler | Used for | Type | This page |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search | AI search | Allowed |
| GPTBot | OpenAI model training | Training | Allowed |
| ChatGPT-User | ChatGPT, when a user asks | User request | n/a |
| Claude-SearchBot | Claude search | AI search | Allowed |
| ClaudeBot | Anthropic model training | Training | Allowed |
| Claude-User | Claude, when a user asks | User request | Allowed |
| PerplexityBot | Perplexity search | AI search | Allowed |
| Perplexity-User | Perplexity, when a user asks | User request | n/a |
| Googlebot | Google Search, AI Overviews and AI Mode | AI search | Allowed |
| Google-Extended | Gemini training (not Search) | Training | Allowed |
| Applebot-Extended | Apple model training | Training | Allowed |
| CCBot | Open dataset used by many models | Training | Allowed |
Blocking training crawlers is a choice and doesn't affect AI search. What each crawler does
No structured data found.
Fix: Add JSON-LD: Organization on the home page, Article with author and dates on articles, Product or SoftwareApplication on product pages.
How to fix it0 of 30 subheadings are phrased as questions.
Fix: Phrase some H2s as the questions people ask, with a one or two sentence answer right below each.
How to fix itNo machine-readable date or author found. Assistants use dates and authorship to judge whether a page is current and who stands behind it.
Fix: Show "Updated [date]" and the author, and add datePublished, dateModified and author to the Article JSON-LD.
How to fix itNo /llms.txt. It is optional, and no major assistant has said it uses one, but it is cheap to add.
Fix: Add a short Markdown summary of the site with links to its key pages at /llms.txt.
How to fix itEvery AI search and answer crawler is allowed.
How to fix it718 words are in the HTML before any JavaScript runs.
How to fix itReturns HTTP 200 with no noindex or nosnippet rules.
How to fix it“Astro”
robots.txt points to a sitemap.
It scores 77 out of 100. One problem stops assistants from reading or quoting this page. Fix it first.
None. Its robots.txt lets in every AI crawler this check looks at.
It was last checked on October 30, 2026 and is re-checked once a day.
This check shows whether they can. Surfacepath Pro will ask them your customers' questions every week and show whether you're mentioned, cited, or replaced by a competitor.