People used to search for “software agency for X” and click through a page of blue links. Now more of them ask ChatGPT, Claude, or Perplexity the same question and read one answer back. If a business isn’t in that answer, it doesn’t exist for that buyer, no matter how good the work is. We built dawloom.com around that fact, and none of it is theory: view source on this page, fetch our robots.txt, or ask an assistant about Dawloom and see what it finds.
Let the crawlers in, by name
Our robots.txt names crawlers individually and allows each one: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, and Bingbot. That list looks redundant until you notice the names split into three different jobs.
GPTBot and ClaudeBot crawl in the background to train models. OpenAI is explicit that disallowing GPTBot “indicates a site’s content should not be used in training generative AI foundation models,” which also means blocking it only affects some future model’s training data. It doesn’t touch what an assistant can find today. OAI-SearchBot and PerplexityBot do a different job: they crawl to build the index an assistant searches when it answers a live question, the same role Bingbot and Google-Extended play for search results.
Then there’s a third kind. ChatGPT-User, Claude-User, and Perplexity-User fetch one specific page, live, because a real person’s question needs it right then. OpenAI describes ChatGPT-User as “not used for crawling the web in an automatic fashion” since “these actions are initiated by a user.” Perplexity’s docs go further and note that Perplexity-User “generally ignores robots.txt rules” for the same reason: a person asked, not a bot browsing on its own schedule. A Disallow line written to keep training crawlers out won’t stop that kind of live fetch. Treating every bot name the same is how a site ends up blocking exactly the traffic that would have recommended it.
Give the crawler something worth reading
Naming the right bots doesn’t help if the page they land on is an empty div waiting for JavaScript to fill it in. We covered this in detail in why this site loads in under a second: dawloom.com is built with Astro and ships as static HTML, so what a crawler fetches is already the finished page, with no browser required to render it first.
Structured data instead of scraped guesses
Every page here carries an Organization block in schema.org JSON-LD format. FAQ sections carry FAQPage schema, and service pages carry Service schema. None of it is decorative copy for search engines to skim; it’s the same object the layout renders.
const organization = {
'@context': 'https://schema.org',
'@type': 'Organization',
name: 'Dawloom',
url: 'https://dawloom.com',
description:
'Software development agency building mobile apps, web platforms, AI integrations, and AI SaaS products for clients worldwide.',
address: { '@type': 'PostalAddress', addressCountry: 'PK' },
};An assistant reading this doesn’t have to infer what Dawloom does from surrounding marketing copy. The fact is marked up as a fact, in a format both Google and every major AI crawler already parse. Service pages carry the same treatment, with a Service block that names the provider as the same Organization, so a page about mobile app development and the homepage agree with each other instead of leaving an assistant to reconcile two descriptions of the same company.
A plain-text map, and markdown on request
We also publish an llms.txt, a short plain-text index of the site an assistant can read in one request instead of crawling every page to work out what’s here. And if a request arrives asking for Accept: text/markdown, our Cloudflare Worker serves the markdown source of that page instead of HTML, with a Content-Signal: ai-train=yes, search=yes, ai-input=yes header attached, stating plainly that the content is fine to train on, search, and pull into a model’s context.
Watching what the crawlers actually ask for
The same Worker logs every AI crawler visit to PostHog: which page, which bot, what status code came back. When GPTBot or ClaudeBot requests a page that returns a 404, that’s a machine telling us what it expected to exist here. We treat that list as a content backlog, pages assistants are already looking for that we haven’t written yet, rather than a list of pages we guessed people might want.
What none of this promises
Getting the mechanical layer right makes a site legible to a crawler. It does not guarantee a citation, a ranking, or a mention in any specific answer, and we haven’t measured a lift from any of it because we haven’t run that experiment. Anyone claiming a precise percentage increase in AI citations from a robots.txt change is telling you a number nobody has verified yet; the field is too new and too dependent on each assistant’s internal ranking to support that kind of certainty. What we can say is narrower and still true: an assistant can’t recommend what it can’t read, and most business sites fail that test before content quality ever becomes the question. Fixing robots.txt, rendering static HTML, marking up facts, and publishing an llms.txt is cheap relative to everything that comes after it. That’s why we did it before we did anything else on this site.
The verdict
If a client asks whether the mechanical layer is worth the engineering time, the answer is yes, and it’s a short project you build once and leave alone. Static HTML and a JSON-LD block don’t need the weekly attention a content calendar does. What none of it replaces is having something worth citing in the first place, which is a harder problem than robots.txt. If you want your own site or product checked against this same list, from crawler access to structured data to how AI features get built on top of it, see how we build or tell us what you’re working with and we’ll tell you honestly what’s missing. For the AI work itself rather than the plumbing around it, our AI services cover the rest.