Insights · AI visibility
What is llms.txt, and does your business need one?
llms.txt is a proposed file that summarises your website for AI models. What goes in it, how it differs from robots.txt, and how much it matters.
llms.txt is a plain text file, written in Markdown, that sits at the root of your website and summarises what your business is and which pages matter. It was proposed in September 2024 as a way to help language models read a site quickly. It is cheap to add, but no major AI engine has confirmed it uses it, so it should come after reviews, content and citations, not before.
Where did llms.txt come from?
Jeremy Howard of Answer.AI published the proposal in September 2024, at llmstxt.org. The idea is simple. Websites are built for people, with menus, banners, cookie notices and scripts. A language model reading your site has to dig through all of that to find the facts. An llms.txt file gives it a short, clean summary in one place: yourdomain.com/llms.txt.
It is a proposal, not an agreed standard. Nobody has to support it, and no search or AI company has committed to reading it.
What goes in an llms.txt file?
The proposal sets out a loose structure, all in Markdown:
- A heading with your business name. This is the only required part.
- A short summary in a blockquote. One or two sentences on what you do and for whom.
- Optional detail. A few lines or bullet points with key facts.
- Sections of links. Headings such as “Services” or “Guides”, each with a list of links to your most useful pages and a note on what each one covers.
Here is a short example for a made-up Nerja dental clinic. The name and links are illustrative.
# Clínica Dental Ejemplo
> Clínica Dental Ejemplo is a private dental clinic in Nerja, Spain.
> We treat local and international patients in English, Spanish and German.
- Where: Nerja and Torrox, Costa del Sol
- Languages: English, Spanish, German
- Contact: [email protected]
## Services
- [Dental implants](https://www.example.com/implants/): how treatment works, timings and what to expect
- [Emergency appointments](https://www.example.com/emergency/): same-day slots and how to book
## Patient guides
- [Treatment for visitors](https://www.example.com/visitors/): planning treatment around a holiday or a short stay
That is the whole idea. A clear name, a clear summary, and a short map of your best pages.
How is llms.txt different from robots.txt and a sitemap?
They do three different jobs.
- robots.txt tells crawlers what they may and may not fetch. It is about permission. It has been a web convention for decades and the major crawlers respect it.
- A sitemap lists every page you want search engines to find. It is about completeness. Google and Bing both use sitemaps.
- llms.txt summarises what your site is about and points to the pages that matter most. It is about understanding. It is also the only one of the three that the big engines have not confirmed they use.
So llms.txt does not replace either of the others. If your robots.txt blocks an AI crawler, a perfect llms.txt makes no difference, because the crawler is not allowed in to read it.
Is your robots.txt blocking AI crawlers?
This is the more important check, and many businesses never make it. Some website platforms, security plugins and hosting services can block AI crawlers by default. Others were blocked on purpose a year or two ago and nobody remembers why.
The crawlers worth knowing about:
- OAI-SearchBot is the crawler OpenAI uses for ChatGPT’s search results. If it is blocked, OpenAI says your pages will not be shown in ChatGPT search answers.
- GPTBot is OpenAI’s crawler for training its models. Blocking it is a separate choice from blocking OAI-SearchBot.
- PerplexityBot is the crawler Perplexity uses to find and link to websites in its answers.
- Google-Extended is not a separate crawler. It is a control token in robots.txt that tells Google whether content it has already crawled can be used to train its Gemini models and to ground answers in Gemini apps. Google says it does not affect your inclusion or ranking in Google Search. Blocking it is a training decision, not a search one.
To check, open yourdomain.com/robots.txt in a browser. Look for lines that say Disallow: / under any of those names, or under User-agent: *. If you see one and did not mean to, ask whoever manages your site to change it.
There is a fair business decision here. Some publishers block training crawlers on principle. For a Costa del Sol clinic, school or estate agency that wants to be recommended, blocking the search crawlers is almost always a mistake.
Should you add an llms.txt file?
Yes, if it takes your web developer half an hour. No, if it comes at the expense of anything else.
Our honest view: llms.txt costs almost nothing and may help in future, if engines start reading it. Today there is no public evidence that ChatGPT, Gemini or Perplexity use it when deciding who to name. What does move AI answers is well documented: specific content and independent sources that get cited, consistent descriptions across the web, and reviews that mention what you actually do. Clear schema markup sits in the same category as llms.txt: a solid foundation that is quick to get right.
A worked example: this site
This website publishes its own llms.txt at /llms.txt. It opens with our name and a one-paragraph summary of what we do and where. It then lists our services and prices, our key pages, the industries and areas we cover, every article on the site, and links to the Spanish pages. The file is generated from the same content as the site each time it is built, so it never falls out of date when we add a page.
We also allow the main AI crawlers in our robots.txt, including GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended, because being read by them is the point.
What to do this week
- Open your robots.txt and check nothing important is blocked.
- If your site has no llms.txt, add a short one like the example above.
- Then spend the rest of your time on the things that matter more. Our guides to Google AI Overviews and GEO are good places to start.
If you want to see what ChatGPT, Gemini and Perplexity say about your business today, our free AI visibility check tests ten real questions and compares you with three competitors.