llms.txt and AI Crawler Access

The AI Crawler Landscape

Several categories of bot now crawl for AI purposes, and they do different things:

• Training crawlers collect content to train future models. The content may inform a model permanently with no ongoing attribution or traffic.
• Retrieval crawlers fetch pages in real time to answer a specific user question, usually with a citation and link.
• Search index crawlers build indexes that AI products query.

These warrant different decisions. Blocking retrieval crawlers removes you from AI answers that would have cited you. Blocking training crawlers does not affect current answers at all.

Controlling Access via robots.txt

Most major AI crawlers respect robots.txt and publish their user agent strings. Directives are set the same way as for any bot — a User-agent block with Allow or Disallow rules.

Publishers commonly distinguish between the two cases: permitting retrieval crawlers so their content can be cited in answers, while restricting training crawlers. Because vendors use separate user agents for these functions, the distinction is enforceable.

Check your server logs for what is actually crawling you. The list of active AI user agents changes, and rules written a year ago are probably incomplete.

What llms.txt Proposes

llms.txt is a proposed convention: a markdown file at the site root that gives language models a curated, clean map of the site — the key pages, what they cover, and links to plain-text versions.

The reasoning is that HTML pages carry navigation, ads and boilerplate that waste context and obscure the substance. A curated file offers the important content directly.

Be clear-eyed about status. This is a community proposal, not a standard any major AI vendor has committed to honouring. It is cheap to implement and currently speculative in benefit. Implement it as a low-cost bet, not as a ranking tactic.

Implementing llms.txt

The format is plain markdown: an H1 with the site name, a short blockquote summary, then sections of annotated links. Keep it curated rather than exhaustive — a full sitemap defeats the purpose.

Some sites also publish llms-full.txt containing the actual content rather than links, for cases where a model can ingest the whole corpus.

Neither replaces sitemap.xml or robots.txt. They are additive.

Making the Decision

Arguments for allowing AI crawlers: visibility in AI answers, citation-driven brand awareness, referral traffic from retrieval, and presence in the tools people increasingly start with.

Arguments against: content used without compensation, potential substitution for visits, and loss of control over how material is represented.

Most publishers whose business depends on being found conclude that retrieval access is worth permitting. Publishers whose business is the content itself — subscription journalism, paid research — more often restrict. There is no universally correct answer, and the decision should be made deliberately rather than by default.

Deciding Your Crawler Policy Deliberately

Crawler access is a business decision presented as a technical one, and most sites have not made it — they have inherited a default.

The positions available, and who each suits:

Open to everything. Maximum visibility, no control over training use. Suits anyone whose content is marketing rather than product.

Allow retrieval, block training, where the distinction is respected. Several crawlers are separable — one fetches to answer a live question, another collects for training — and allowing the first while blocking the second is a coherent position. It depends on voluntary compliance.

Block everything. Protects the material and removes you from the answers, including the ones where being named would have helped.

The decision belongs to whoever owns the commercial model, not to whoever edits `robots.txt`. Write down what you chose and why, because the person who inherits the file cannot infer the intent from the syntax.

What llms.txt Is and Is Not

The proposal is a plain-text file at the site root that points to the pages you consider authoritative, in a form easy to read without rendering.

What it is: a curation signal — here is the canonical version, here is the current documentation, ignore the archived variants. For a site with a large surface and a small authoritative core, that is a genuinely useful thing to state.

What it is not: a standard anyone is obliged to honour, a ranking mechanism, or a substitute for `robots.txt`. It grants no access and removes none; it is advisory to whoever chooses to read it.

Is it worth doing? It costs an afternoon, it cannot hurt, and adoption is uneven enough that the honest expectation is little measurable effect. That is a reasonable basis for doing it and a poor basis for a project plan.

The one caution: like every other derived file on a site, it goes stale. An `llms.txt` pointing at pages that have moved is worse than none — so generate it rather than writing it, and regenerate it when the site changes.

Sources

What each claim on this page rests on. Entries are typed so you can see which are primary.