AI crawler checker: can ChatGPT read your website?

To see if ChatGPT can read your website, paste a page address into the checker below. It tells you whether OAI-SearchBot and ten other AI and search crawlers are allowed by your robots.txt, whether your firewall lets them through, and whether your words are in the HTML or only appear after JavaScript runs.

Who this is for: anyone who wants their site to show up in ChatGPT, Perplexity, Claude, Copilot, or Google AI Overviews and isn’t sure the crawlers can get in. Who should use something else: if you want to know whether an assistant already names you, no checker can answer that. Run your buyers’ questions in the assistants themselves and count the mentions. Our ChatGPT recommendation playbook covers that side.

The check runs on our Cloudflare Worker, because browsers can’t read most sites’ robots.txt files from another site. It accepts public https pages only. It refuses private, loopback, and cloud metadata addresses, checks DNS answers and every redirect against the same list, sends no cookies, stops after 8 seconds, and reads at most 1 MB per response. It stores nothing. The tool page has a full example report as plain HTML.

What it checks that robots.txt checkers miss

We searched “is my website visible to ChatGPT” and “AI crawler checker” on 2026-10-13 and tried the top tools. Most read robots.txt and stop there. That’s the first of four gates a crawler has to pass.

1. robots.txt, per crawler. It reads the file the way RFC 9309 says to. The group named for the crawler wins. Without one, the * group applies. Inside a group, “The most specific match found MUST be used”, and an Allow wins a tie. It checks the exact page you entered, not only /.
2. The firewall. It fetches the page twice: once as a normal visitor, once with a user agent that carries the OAI-SearchBot token. If the first gets a 200 and the second gets a 403 or a challenge page, a bot setting is blocking AI crawlers that robots.txt allows.
3. Text without JavaScript. It counts the words in the raw HTML, with scripts and styles removed. Under 50 words fails. Under 200 warns.
4. Indexing controls. It reads the robots meta tag and the X-Robots-Tag header for noindex, none, nosnippet, and max-snippet:0.

It also reports the sitemap, Bing Webmaster verification, llms.txt, and whether Accept: text/markdown returns markdown. The last two are graded as info, never as failures, for reasons below.

Real results from seven sites

We ran the checker against our own site and six well-known ones on 2026-10-13, from a local copy of the checker. Each homepage was checked once. Sites change their robots.txt, so run them again before you quote this table.

Three things stood out.

• The news sites block AI search on purpose. BBC, CNN, and the New York Times all disallow OAI-SearchBot and PerplexityBot, and all three still allow Googlebot and Bingbot. That’s a business choice, not a mistake. BBC doesn’t block Claude-SearchBot or Claude-User but does block ClaudeBot, so it separates Anthropic’s search from Anthropic’s training. CNN and the Times block all three.
• A blocked checker is a signal too. The New York Times answered our checker with HTTP 403 for both the normal request and the one carrying the OAI-SearchBot token. It answers real browsers. Bot protection that turns away unknown visitors often turns away crawlers you want, so the report flags it.
• A single-page app is close to empty. Excalidraw is a drawing app, so this is expected. Its homepage HTML holds one word, an empty <div id="root">, and six script tags. Its robots.txt allows every crawler on the homepage, so a robots.txt-only checker would give it a clean pass. A crawler that doesn’t run JavaScript gets nothing to read.

Each check, and how to fix a failure

1. AI search crawlers in robots.txt

Each AI company runs separate crawlers for search, for pages a person asks about, and for training. Blocking the training crawler doesn’t hide you from AI search. Blocking the search crawler does.

Sources: OpenAI’s crawler page, Perplexity’s, Anthropic’s, and Google’s, all read 2026-10-13.

OpenAI is blunt about the stakes: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” It also says a robots.txt change can take “~24 hours” to reach its search systems. Google says Google-Extended “does not impact a site’s inclusion in Google Search”, but it does cover grounding, which it defines as “providing content from the Google Search index to the model at prompt time”. So blocking Google-Extended can keep you out of Gemini answers even though Google Search still has you.

The user fetchers are a special case. OpenAI says “robots.txt rules may not apply” to ChatGPT-User, and Perplexity says its user fetcher “generally ignores robots.txt rules”. Anthropic says blocking Claude-User “prevents our system from retrieving your content in response to a user query”. The checker warns when these are blocked, because the result depends on the company.

Common failure: a 2024 “block all AI” template that lists every OpenAI token, including OAI-SearchBot. Fix: allow the search and user crawlers by name, and block only the training tokens you object to:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

One more trap. If robots.txt returns a 5xx error, RFC 9309 says the crawler “MUST assume complete disallow”. A missing file (a 4xx) means the opposite: the crawler “MAY access any resources on the server”. The checker fails the first and reports the second as info.

2. Firewall and CDN

robots.txt is a request. A firewall is a wall. If your CDN or security plugin answers AI crawlers with a 403, a 429, or a challenge page, they never see your content, whatever robots.txt says. Cloudflare announced on 2025-07-01 that it was “changing the default to block AI crawlers unless they pay creators for their content” (Cloudflare blog).

The checker sends one request whose user agent carries the OAI-SearchBot token, followed by DawloomCrawlerCheck and the words “test request, not from OpenAI”. Rules that match on the user agent fire on it. Rules that verify a crawler by its IP address will treat it as an ordinary visitor, because it doesn’t come from OpenAI’s IP ranges. So a pass here doesn’t prove the real crawler gets through, and a fail almost always means a rule is blocking it.

Fix: in your CDN’s bot settings, allow verified AI search crawlers, and check your security plugin’s bot list. Then run the check again.

3. Text without JavaScript

This is the gate most sites never test. Vercel and MERJ studied crawler traffic across Vercel’s network and found that “none of the major AI crawlers currently render JavaScript” (Vercel, December 2024, measured by the vendor). The crawlers they named include OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, and PerplexityBot. Gemini is the exception, because it uses Googlebot’s rendering. That study is close to two years old, so crawler behaviour may have moved since. Server-rendered HTML works either way.

The checker strips scripts, styles, and markup from the raw HTML and counts what’s left. It also looks for an empty app root, such as <div id="root"></div>, which is the sign of a client-rendered app.

Fix: render the content on the server, or build the page to static HTML. Next.js, Astro, SvelteKit, and Nuxt all do this when you use their server rendering or static output. We explain the trade-offs for buyers in static, server-rendered, or islands. This site ships 1,372 words of HTML on the homepage, and why this site loads in under a second shows how.

4. noindex and nosnippet

Google’s guide to AI features says a page “must be indexed and eligible to be shown in Google Search with a snippet” to appear as a supporting link in AI Overviews or AI Mode (Google Search Central). It lists nosnippet, data-nosnippet, max-snippet, and noindex as the controls that limit what’s shown. So noindex fails, and nosnippet or max-snippet:0 warns. The checker ignores robots tags on a page that didn’t return 200, because error and challenge pages often carry noindex themselves.

5. Sitemap and Bing

ChatGPT, Copilot, and many other assistants pull from search indexes before they answer. Microsoft’s Bing Webmaster Tools added an AI Performance report on 2026-02-10 that shows “how publisher content appears across Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations” (Bing Webmaster blog). To get that report you need a verified site.

The checker looks for a sitemap listed in robots.txt or at /sitemap.xml, and for the two Bing verification methods it can see from outside: the msvalidate.01 meta tag and the BingSiteAuth.xml file. DNS verification is invisible to any outside checker, so a missing tag is info, not a warning.

6. llms.txt and markdown

These two are optional, and the report says so. Google’s AI features guide says “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” We found no major AI search engine that says it reads llms.txt.

We serve both anyway, because they’re cheap and agents that do ask for them get a cleaner answer. Our llms.txt is 16,942 bytes, and any page on this site returns markdown when the request sends Accept: text/markdown. The checker warns only when /llms.txt answers with an HTML page, which usually means a single-page app’s fallback route is catching it.

Use it from a script or an agent

The same check is a free API. No key, and the limit is 10 checks a minute per IP. A page that fails still returns HTTP 200 with a failing report. A 4xx means the address itself was refused, for example not_https or blocked_host.

curl -X POST https://dawloom.com/api/v1/crawler-check \
  -H "Content-Type: application/json" \
  -d '{"url":"https://dawloom.com/"}'

The schema is in our OpenAPI document under checkAiCrawlerReadiness. We built it on the same Worker and the same address checks as our MCP server checker.

FAQ

How do I know if ChatGPT can see my website?

Check three things: that robots.txt allows OAI-SearchBot, that your firewall doesn’t block it, and that your text is in the HTML. The checker above does all three. Then search ChatGPT for your brand or a page title. If it can cite the page, it can see it.

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI lets you set the two separately. Block GPTBot and allow OAI-SearchBot if you want search without training.

Why is my website not showing up in ChatGPT even though crawlers are allowed?

Access is only the first step. Assistants name the sites that other sites mention, and they prefer pages that answer the question better than what they already cite. A site that passes every check here can still go unmentioned.

Do AI crawlers read JavaScript?

Most don’t. A 2024 study by Vercel and MERJ found that none of the major AI crawlers rendered JavaScript, with Gemini (through Googlebot) as the exception. Put your main content in the HTML the server sends.

Is llms.txt required to appear in AI answers?

No. Google says you don’t need AI text files for AI Overviews or AI Mode, and we found no major AI search engine that says it reads llms.txt. It’s a cheap extra, not a requirement.

If your site fails the JavaScript gate and you need it rebuilt with server rendering, that’s the kind of web development work we scope in writing, billed hourly or by milestone.