What Is llms.txt? A Practical Guide to the AI Website Index

An llms.txt file is a small Markdown document that gives AI systems a curated route through your website. Instead of asking a model to infer what matters from a large sitemap, you provide the important pages, grouped by purpose, with short descriptions that explain why each link is useful.

The file is useful when your site has product pages, documentation, support content, pricing, or articles that benefit from a clear reading order. It does not replace a sitemap or robots.txt; it adds a human-edited layer of context for language models and AI retrieval workflows.

Think of llms.txt as an editorial map, not a crawler directive. It helps an AI system decide what to read first and what each page is for.

What goes in an llms.txt file?

The proposed format uses ordinary Markdown. Start with a single H1 containing your site or project name. You can follow it with a short blockquote description, then group useful links under H2 headings such as Product, Documentation, Support, or Company.

Each link can carry a concise description after a colon. That description is valuable because a URL alone rarely tells a model whether the page contains setup instructions, a pricing explanation, or a policy.

# Example Product

> A clear map of the product, documentation, and support resources.

## Documentation

- [Quickstart](https://example.com/docs): Install and configure the product.
- [API reference](https://example.com/api): Endpoint and authentication details.

File	Primary job	Best for
sitemap.xml	Lists crawlable URLs	Search engine discovery
robots.txt	Sets crawler rules	Crawl control
llms.txt	Curates high-value pages	AI understanding and retrieval

How is llms.txt different from a sitemap?

A sitemap aims for coverage. It can list every canonical URL, even when many pages are thin, duplicated, or unimportant to an AI assistant. An llms.txt file aims for clarity: it points to the pages that explain your product, show people how to use it, and answer recurring questions.

That difference is why page grouping and descriptions matter. A flat list shifts the interpretation work to the AI. A structured file lets you state your priorities explicitly.

How do you create an llms.txt file?

Start with the pages you would give to a new customer, support teammate, or technical writer. Include your homepage, core product pages, documentation, pricing, key policies, and the most useful articles. Then remove low-value utility pages and add plain-language descriptions.

You can create the first draft with the free llms.txt generator, which turns a sitemap, homepage, or manual URL list into an editable index. Before publishing, run the result through the llms.txt validator to catch missing descriptions, weak structure, and link issues.

Where should you publish it?

Publish the completed file at your website root: https://example.com/llms.txt. Keeping the route predictable makes it easy for people and tools to locate. Review it whenever you ship a new product area, replace documentation, or reorganize your information architecture.

If your AI workflow needs the full readable content rather than a compact route map, use an llms-full.txt knowledge file alongside the index. The two formats solve different problems: one helps discovery; the other supplies deeper context.