llms.txt Explained: What It Is and Whether You Need One
llms.txt is a proposed markdown file at your site's root that gives AI systems a short, curated summary of your key content, similar in spirit to robots.txt or sitemap.xml, but for language models.
Proposed by Jeremy Howard (Answer.AI) in September 2024, llms.txt has real but limited adoption today: thousands of sites publish one, and OpenAI, Anthropic and Google publish files for their own developer docs, but it is not yet a requirement any major AI crawler is confirmed to follow.
Score Your PageWhat Is llms.txt?
llms.txt is a proposed standard for a plain markdown file, placed at /llms.txt on a site's root (or any subpath), that gives an AI agent a short, structured overview of a site or a section of it: what the site is, and a curated list of links to the pages worth reading. The idea is to solve a real problem: a normal HTML page is built for people, wrapped in navigation, ads and JavaScript, and converting it back into clean text an AI model can use efficiently is wasteful and imprecise.
It follows the naming convention of robots.txt and sitemap.xml, a fixed filename at a predictable location, but it does a different job than either. robots.txt tells automated tools what they're allowed to access. sitemap.xml lists every indexable page for search engines. llms.txt instead offers a short, curated, human-and-machine-readable summary meant to be read on demand, when an AI agent needs to understand a site or find something specific in it, not crawled wholesale.
Adoption is real but not universal, and it is important to be precise about what kind of adoption exists. Thousands of sites now publish an llms.txt file, some documentation platforms generate one automatically, and Chrome's Lighthouse tool audits for one as part of its agentic-browsing checks. OpenAI, Anthropic and Google Gemini all publish llms.txt files for their own developer documentation. What is not yet confirmed is that the general-purpose AI crawlers used for everyday web browsing and training, GPTBot, ClaudeBot, PerplexityBot and similar, actually fetch and prioritise llms.txt content the way search engines reliably honour robots.txt. The clearest confirmed use case today is coding agents and documentation tools, which do follow llms.txt files to find API references and tutorials.
What llms.txt Does and Does Not Do
A Curated Summary, Not a Full Crawl
Unlike sitemap.xml, which lists everything, llms.txt is meant to hold only what an AI agent actually needs: a short description plus links to the pages that matter most.
Not a Confirmed Ranking or Citation Signal
No major AI company has confirmed that publishing an llms.txt file improves whether your content gets cited. Treat it as a convenience file for agents that choose to read it, not a growth lever on its own.
A Complement to robots.txt and sitemap.xml, Not a Replacement
llms.txt doesn't replace either file. robots.txt still controls crawler access, and sitemap.xml still lists every page for search engines. llms.txt adds a curated, AI-readable layer on top.
Should You Create One?
Low Effort to Create
A minimal llms.txt is a short markdown file: a title, a one-line summary, and a list of links to your most important pages. It takes far less time than most other GEO work.
A Reasonable Bet on Where Things Are Heading
Adoption has grown from a 2024 proposal to thousands of publishing sites and support from Chrome's Lighthouse in under two years. Publishing one now costs little and positions you ahead of any future crawler support.
Strongest Case for Documentation and API Sites
The clearest, confirmed use case today is coding agents reading developer documentation. If you run docs, an API reference, or a developer portal, llms.txt has the most proven payoff right now.
Create Your llms.txt in 3 Steps
List Your Most Important Pages
Pick the pages you'd want an AI agent to read first: your homepage, key product or docs pages, and anything that explains what you do.
Write the File
Start with an H1 title and a one-line blockquote summary, then add a markdown list of links under one or more H2 sections. Keep it short. The whole point is that it fits easily in a model's context.
Publish at /llms.txt and Recheck Your GEO Score
Place the file at your site's root. Then make sure the pages it links to are themselves crawlable, well-structured and citable. llms.txt only points an agent toward your content, it doesn't fix what the content lacks.
llms.txt FAQ
What is llms.txt?
llms.txt is a proposed markdown file placed at a site's root (/llms.txt) that gives AI agents a short, curated summary of the site plus links to its most important pages. It was proposed by Jeremy Howard of Answer.AI in September 2024.
Do ChatGPT, Claude and Perplexity actually use llms.txt?
Not confirmed as a general rule. OpenAI, Anthropic and Google publish llms.txt files for their own developer documentation, and coding agents reliably follow llms.txt files to find API references. But no major AI company has confirmed that its general-purpose crawler (GPTBot, ClaudeBot, PerplexityBot) fetches and prioritises an arbitrary site's llms.txt during normal browsing or training. Treat it as emerging and optional, not a confirmed requirement.
How is llms.txt different from robots.txt?
robots.txt tells crawlers what they're allowed to access and is a well-established, widely honoured standard. llms.txt instead offers a curated, human-and-AI-readable summary of a site's key content, meant to be read on demand rather than crawled. They serve different purposes and are meant to coexist.
Isn't this the same as sitemap.xml?
No. A sitemap lists every indexable page, mainly for search engines, and rarely includes AI-friendly markdown versions of pages. llms.txt is deliberately short and curated, a handful of the pages that matter most, not an exhaustive list.
Should I create an llms.txt file for my site?
It's a low-effort, low-risk addition, especially if you run documentation or an API reference, where the confirmed use case (coding agents) already exists. For a general marketing site, treat it as a small, forward-looking addition rather than a priority over crawlability, structured data and content clarity, which have a proven effect on AI visibility today.
What does a valid llms.txt file need to contain?
An H1 with the site or project name is the only required section. Typically it's followed by a one-line blockquote summary, optional context paragraphs, and one or more H2-headed sections containing a markdown list of links, each with a short description.
Check Whether Your Site Is Ready to Be Cited
An llms.txt file only helps if the pages behind it are actually crawlable and citable. Score your page on 22 metrics first. Five checks per domain are free every 30 days.
Score Your Page