llms.txt, what it does and what it does not do

llms.txt is a Markdown file at the root of a site that tells a language model what the site is and where the important pages are. It was proposed in September 2024. Two years on, it is one of the checks Moonleap runs, and also one of the checks we describe most carefully, because the honest answer to "does it help my AI visibility" is "not in the way people hope".

What is llms.txt supposed to do?

The proposal is small: an H1 with the site name, a blockquote summary, then sections of links with one-line descriptions. An optional llms-full.txt holds the full text of the important pages. The idea is that a model with a limited context window reads one clean file rather than crawling a hundred templated pages.

Ours is at moonleap.io/llms.txt. It lists the product pages, the MCP endpoint and the one-line command to connect, every blog post with its summary, and three facts about how Moonleap handles data. It is generated from the same source as the blog, so it never goes stale.

Which AI systems actually read it?

The systems that decide whether you appear in an AI answer are Google's AI Overviews and AI Mode, ChatGPT search, Perplexity and Gemini. Google's documentation says AI Overviews draw on the ordinary Googlebot index; there is no mention of llms.txt. OpenAI documents three crawlers (GPTBot for training, OAI-SearchBot for search, ChatGPT-User for browsing on a user's behalf) and how to allow or block them in robots.txt; none of the documentation references the file. Anthropic and Perplexity are in the same position.

So a site with a perfect llms.txt and a robots.txt that blocks OAI-SearchBot is invisible to ChatGPT search. The order of work is: crawler access first, then indexable content, then structured data, then this file.

Then why keep the check at all?

When a Claude Code or Cursor session is asked "what does moonleap.io do", the agent can fetch /llms.txt and get the answer without parsing a landing page. That is a real, if narrow, use. The exercise of writing the summary is the second use: most sites cannot state what they are for in one blockquote, and the failure shows up in every AI answer about them.

Moonleap raises the missing file as a low-severity finding, never as a reason your AI visibility is poor. The reasons for that live elsewhere: whether the answer engines can crawl you, whether your pages answer the questions people ask them, whether anyone else on the web says what you say about yourself.

What is the mistake to avoid?

We have seen llms.txt files with two hundred links, marketing paragraphs and keyword lists. Nothing reads them that way. Keep it to the pages a careful reader would want, one honest line each, and regenerate it when the site changes. Ours is under two kilobytes.

FAQ

Should I also publish llms-full.txt?

Only if your important pages are long and templated (documentation sites, mostly). For a marketing site the short file is enough.

Does llms.txt replace robots.txt or the sitemap?

No. Crawlers follow robots.txt; search engines discover pages through the sitemap and links. llms.txt is documentation, not a directive.

Will Google penalise a site for having one?

There is no evidence of that. It is a plain text file like any other.