llms.txt, robots.txt and the AI crawlers you should allow

robots.txt: check what you're blocking

Many sites inherited a blanket disallow from a staging config, or a security plugin that blocks unknown user agents. AI crawlers use their own agent strings, and a rule aimed at scrapers can catch them too. Decide deliberately which you want reading you, then write explicit rules rather than relying on the default.

• GPTBot and OAI-SearchBot — OpenAI training and search crawlers.
• ClaudeBot — Anthropic's crawler.
• Google-Extended — controls Gemini and AI Overviews use, separate from Googlebot.
• PerplexityBot — Perplexity's retrieval crawler.
• CCBot — Common Crawl, which feeds many downstream datasets.

llms.txt: a briefing note for machines

llms.txt is a plain markdown file at your root that says who you are, what you do, and links the pages you want used as sources. Think of it as the one-page brief you'd hand a new analyst. It is a convention rather than a standard, so treat it as low-cost insurance: it takes twenty minutes and it cannot hurt.

Keep it short, factual and link-heavy. Name your product, your categories, your pricing page and your documentation. Avoid adjectives entirely.

The rest of the root-level checklist

While you're in there, confirm the supporting files exist and are correct.

• sitemap.xml listing every indexable page, referenced from robots.txt.
• A real 404 status code on missing pages, not a 200 with an error message.
• HTTPS with an HTTP redirect and HSTS enabled.
• security.txt if you handle any customer data.

Verify, don't assume

Request each file with a plain HTTP client rather than a browser — a CDN rule or bot filter can serve a browser fine and a crawler a challenge page. Beacon's grader fetches them the same way a crawler would and shows you exactly what came back.