Check if ChatGPT and AI crawlers can read your site
Paste a page address. The checker reads it the way AI crawlers do and tells you which ones may read it, whether a firewall blocks them, and whether your text is there without JavaScript.
Example report: dawloom.com
A real run against our own homepage, checked 2026-10-13 with a local copy of the checker. Grade: pass, in 700 ms.
| Result | Check | What the checker saw |
|---|---|---|
| pass | Page answers | HTTP 200. |
| pass | robots.txt | Found, 10 groups, 379 bytes. |
| pass | AI search crawlers | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, and Bingbot may all read this page. |
| pass | User-triggered fetchers | ChatGPT-User, Perplexity-User, and Claude-User are allowed. |
| info | Training crawlers | GPTBot, ClaudeBot, and Google-Extended are allowed. Your pages may be used for model training. |
| pass | Firewall and CDN | A request carrying the OAI-SearchBot token got HTTP 200, same as a normal request. |
| pass | Indexing allowed | No noindex or nosnippet in a meta tag or X-Robots-Tag header. |
| pass | Text without JavaScript | 1,372 words readable in the raw HTML, before any JavaScript runs. |
| pass | Title and description | Title: "Dawloom | Mobile apps, web platforms, and AI systems". Meta description present. |
| pass | Sitemap | Valid sitemap listed in robots.txt. |
| info | Bing Webmaster Tools | No meta tag or BingSiteAuth.xml file. A site verified by DNS looks the same from outside. |
| pass | llms.txt | Found, 16,942 bytes, starts with a markdown heading. |
| pass | Markdown on request | Accept: text/markdown returns a markdown version of the page. |
The crawlers it checks
Each company runs separate crawlers for search, for pages a person asks about, and for model training. Blocking a training crawler does not hide you from AI search. Blocking a search crawler does.
| robots.txt token | Company | Type | Feeds |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search | ChatGPT search results |
| ChatGPT-User | OpenAI | User fetch | Pages ChatGPT opens when a person asks |
| GPTBot | OpenAI | Training | OpenAI model training |
| PerplexityBot | Perplexity | Search | Perplexity search results |
| Perplexity-User | Perplexity | User fetch | Pages Perplexity opens when a person asks |
| Claude-SearchBot | Anthropic | Search | Claude search results |
| Claude-User | Anthropic | User fetch | Pages Claude opens when a person asks |
| ClaudeBot | Anthropic | Training | Anthropic model training |
| Googlebot | Search | Google Search, AI Overviews, and AI Mode | |
| Google-Extended | Training | Gemini training and Gemini grounding | |
| Bingbot | Microsoft | Search | Bing, which grounds Copilot answers |
Sources: the crawler pages from
OpenAI,
Perplexity,
Anthropic, and
Google,
read 2026-10-13. robots.txt is evaluated by
RFC 9309:
the group named for the crawler wins, then the * group, and the longest matching rule decides.
For agents and scripts
The same check is a free API. No key. 10 checks a minute per IP. A page that fails still returns 200 with a failing report. A 4xx means the address itself was refused.
curl -X POST https://dawloom.com/api/v1/crawler-check \
-H "Content-Type: application/json" \
-d '{"url":"https://dawloom.com/"}'
Schema: /api/v1/openapi.json
(operation checkAiCrawlerReadiness). Docs: developer portal.
Questions clients ask
Short answers, no sales language. If yours isn't here, ask an engineer directly.
Ask us anythingDoes a pass mean ChatGPT will cite my site?
No. A pass means the crawlers are allowed in and can read the text. Whether an assistant names you depends on whether other sites mention you and whether your page answers the question better than the pages it already cites. No checker can see what an AI company has in its index.
Which crawler matters most for ChatGPT search?
OAI-SearchBot. OpenAI says sites that opt out of it will not be shown in ChatGPT search answers. GPTBot is for model training and can be blocked without leaving ChatGPT search.
Why does the checker send a request that names OAI-SearchBot?
Firewalls and CDN bot settings often block AI crawlers by user agent, so a site can allow OAI-SearchBot in robots.txt and still refuse it. The checker sends one request whose user agent carries the OAI-SearchBot token and also says it is a Dawloom test, then compares the answer with a normal request. Real crawlers come from published IP ranges, so a site that checks IPs may treat them differently.
Do I need an llms.txt file?
It's optional. Google says you don't need AI text files to appear in AI Overviews or AI Mode. We serve one, and the checker reports it, but it is graded as info, not as a failure.
Can I call it from a script or an agent?
Yes. POST {"url":"https://example.com/"} to https://dawloom.com/api/v1/crawler-check. No key. The limit is 10 checks a minute per IP. The schema is in the OpenAPI document under checkAiCrawlerReadiness.
More free tools
- AI model compare Put 2 to 4 GPT, Claude, and Gemini models side by side: price, context window, retirement date, and your monthly cost.
- AI model picker Pick the job, volume, and prompt length. Get three API models to test, each with a monthly cost and a test to run first.
- AI feature cost calculator Pick summarise, extract, chat, or agent, enter users and a model, and see the GPT, Claude, or Gemini cost per user per month.
- AI SaaS margin calculator Cost per user, gross margin per plan, and break-even usage for an AI product, with token prices and Stripe fees.
Need a site that AI crawlers can read?
We build fast, server-rendered sites and web apps, billed $30 an hour or by milestone. This site serves HTML, markdown, and llms.txt to every crawler. An engineer replies, not a sales team.