llms.txt and ai.txt: How to Make Your Brand Visible in AI Search
Chandra Shekhar Maurya
TL;DR: do you need llms.txt in 2026?
Publish an llms.txt file if you want a clean, machine-readable map of your site, but do not expect it to lift your ChatGPT or Gemini visibility on its own. The files that clearly matter are robots.txt rules for AI bots, crawlable pages and clear facts. llms.txt, llms-full.txt and ai.txt are cheap extras.
This guide explains each file, what AI engines actually do with it, and how to build all of them in under an hour. It also gives you an example you can copy, a troubleshooting list and a checklist.
Who this guide is for and how it was put together
This guide is for marketers, founders and developers who want their brand found and quoted by ChatGPT, Gemini, Perplexity and similar engines, and who have been told to "add an llms.txt". I judged each file on four questions:
- Is it a documented standard, or a community proposal?
- Do the engines you care about say they read it?
- How much effort does it take, and what can go wrong?
- Can you measure whether it changed anything?
File names, bot names and engine behaviour were checked against the public specification and vendor documentation in October 2026. Where an engine has not documented its behaviour, this guide says so instead of guessing.
What are llms.txt, llms-full.txt, ai.txt and robots.txt?
They are four different files that solve four different problems. Only one of them is enforced by crawlers today.
| File | What it does | Where it lives | Status in 2026 |
| robots.txt | Allows or blocks named crawlers, including AI bots | /robots.txt | Established; respected by major AI crawlers |
| llms.txt | Curated Markdown index of your best pages | /llms.txt | Community proposal; engine use is unconfirmed |
| llms-full.txt | Full text of key pages in one Markdown file | /llms-full.txt | Optional convention; useful for docs and agents |
| ai.txt | States how AI may use your content | /ai.txt | Informal proposal; no major engine commits to it |
If you only have time for one, fix robots.txt first. If a crawler cannot fetch your pages, no other file will rescue your visibility.
What is llms.txt and what does the spec say?
llms.txt is a proposal published at llmstxt.org in 2024. It asks sites to serve a Markdown file at the root of the domain that tells a language model what the site is and where the best pages are. The reasoning is practical: a model has a limited context window and cannot easily digest a full website with navigation, scripts and banners.
The format is deliberately simple. In order, a file contains:
- An H1 with the site or brand name (the only required element).
- A blockquote with a one- or two-sentence summary of what you do.
- Optional paragraphs of plain context, such as who you serve and how you are organised.
- One or more H2 sections, each holding a list of links in the form
[Page name](URL): short description. - An optional H2 called "Optional" for secondary links a model can skip when space is tight.
The spec also suggests that pages worth reading should have Markdown versions at the same URL with .md added. Few sites do this, and you do not need it to start.
What is llms-full.txt and when do you need it?
llms-full.txt is a companion file that holds the full text of your key pages in one Markdown document, so an agent can load everything in one request instead of crawling link by link. It is not part of the minimal spec, but many documentation platforms generate it.
It makes sense when you have product documentation, API references or long help content that coding assistants and AI agents may need to read whole. It rarely makes sense for a ten-page marketing site. Keep it under a size a model can actually use, and regenerate it whenever the source pages change so the two never disagree.
What is ai.txt and does it do anything?
ai.txt is an informal idea for a robots.txt-style file that states what AI systems may do with your content, usually training and data mining. Several variants exist, and no major answer engine has said it will honour any of them.
That makes ai.txt a statement of intent, not an enforcement tool. If you want to stop a company training on your site, the reliable route is a robots.txt rule for its named crawler, plus terms of use. Publish ai.txt if you like, but never rely on it as your only control.
Which robots.txt rules matter for AI bots?
Robots.txt is the one file that AI crawlers from major vendors say they obey. The key point is that each vendor now runs separate bots for training, search and user-triggered fetching, and you can treat them differently.
- OpenAI: GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a person asks about it.
- Anthropic: ClaudeBot is used for training, Claude-SearchBot supports search quality and Claude-User fetches pages on a user's request.
- Perplexity: PerplexityBot indexes pages for its answers, and Perplexity-User fetches pages when a person asks. Perplexity says user-requested fetches may not follow robots.txt.
- Google: Googlebot is what makes pages eligible for AI Overviews and AI Mode. Google-Extended is a separate control for whether content helps train and ground Gemini models; it does not affect Search.
- Others: Applebot-Extended controls Apple's AI use, and CCBot (Common Crawl) feeds many third-party training sets. Microsoft Copilot leans on Bing's index, so Bingbot access matters.
The decision most brands face is simple. If you want to be recommended in live answers, allow the search and user bots. If you object to training use, block the training bots only. Blocking every AI user agent to "protect content" also removes you from the answers where buyers compare vendors.
Do AI engines actually use llms.txt?
Adoption by the big answer engines is uneven, and the honest answer today is that nobody has proven it moves visibility. None of OpenAI, Google or Anthropic documents llms.txt as a ranking or citation signal, and Google has said its AI features need no special files. Server logs from some sites show AI bots occasionally requesting the file, but that is not the same as using it.
Where it does clearly help is with tools that fetch documentation on demand: coding assistants, AI agents and retrieval systems pointed at a docs site. Developer-focused companies were the first to publish it for exactly that reason.
So the rational position is cheap insurance. It takes an hour, it does no harm if your facts are correct, and it forces you to decide which pages define your brand. What drives recommendations is still the stuff in the guide to getting your brand recommended by ChatGPT and AI engines: consistent facts, strong sources and answer-ready pages.
How do you create an llms.txt file step by step?
You can write one by hand in 30 minutes or use the free LLMs.txt Generator to draft it from your site.
- Pick 10 to 25 pages. Choose the pages that define your brand: home, product or service pages, pricing, about, your best guides, and any comparison or FAQ pages. Leave out login, cart, tag and archive pages.
- Write the H1 and summary. Use your brand name and one factual sentence that says what you are, who you serve and where. Avoid slogans; a model needs a definition.
- Group links into 2 to 5 sections. Typical headings are Product, Pricing, Guides, Company and Optional.
- Add a short description to every link. One plain sentence on what the page answers. Descriptions are where most files fail, so make them specific.
- Save as UTF-8 Markdown named llms.txt and upload it to the domain root so it loads at yourdomain.com/llms.txt.
- Link it. Reference the file in your sitemap process or footer so crawlers can discover it, and keep it out of any robots.txt Disallow rule.
- Review it every quarter. Update links when pages move or pricing changes.
Here is an example for a fictional company. Replace every value with your own facts.
# Example Co
> Example Co is a Bengaluru-based invoicing platform for small Indian businesses. It handles GST invoices, payment reminders and bank reconciliation.
Example Co serves freelancers and companies with up to 200 employees. Pricing is public and billed monthly.
## Product
- [Invoicing](https://example.com/invoicing): Create GST-compliant invoices and send them by email or WhatsApp
- [Reconciliation](https://example.com/reconciliation): Match bank transactions to invoices automatically
## Pricing
- [Plans and pricing](https://example.com/pricing): Current plans, limits and what each plan includes
## Guides
- [GST invoice format](https://example.com/guides/gst-invoice-format): Mandatory fields on an Indian GST invoice, with a sample
## Optional
- [Changelog](https://example.com/changelog): Release notes by month
How do you create llms-full.txt and ai.txt?
For llms-full.txt, export the body text of each page listed in llms.txt as clean Markdown, put the page title and URL above each one, and concatenate them in the same order. Strip navigation, cookie banners and repeated footers. If your CMS can output Markdown, automate the build so it regenerates on every publish.
For ai.txt, keep it short and honest. State what you permit for search and answers, what you do not permit for training, and a contact address. Remember it will not stop a crawler on its own, so mirror every rule you care about in robots.txt.
How do you set robots.txt rules for AI bots?
Use the free Robots.txt Generator or write rules by hand. A common pattern for a brand that wants live-answer visibility but no training use looks like this.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
Two cautions. First, a bot with no specific rule falls back to your User-agent: * block, so check that your general rules do not block the pages you want cited. Second, your CDN or firewall can block AI bots even when robots.txt allows them, so test from outside your network.
What commonly goes wrong?
Most failures are boring and fixable in minutes:
- File in the wrong place. It was uploaded to a CMS media folder, so /llms.txt returns a 404.
- Robots conflict. robots.txt blocks a crawler that your other files try to welcome.
- Stale cache. A CDN serves an old version to bots.
- Wrong content type or login wall. The file returns HTML or requires a session.
- Facts that disagree. Pricing, product names or locations in llms.txt differ from the live pages, which teaches models to doubt you.
- JavaScript-only pages. The linked pages render their content client-side, so a bot fetches an empty shell.
Open each file in a private window, check it returns HTTP 200 as plain text, and compare its facts to your live pages. The free AI Crawlability Checker tests robots.txt rules, rendering and llms.txt in one run.
How do you measure whether any of this worked?
Do not judge it by whether a file exists. Judge it by whether your brand appears more often, and more accurately, in the answers buyers see. Build a fixed set of 30 to 50 buyer prompts, record which brands each engine names and which sources it cites, make the changes, and re-run the same prompts weekly.
That is the job of an AI visibility tracker. Listable Labs tracks ChatGPT, Google Gemini and Google AI Mode on every plan from $89/month, with Perplexity, Google AI Overviews and Microsoft Copilot as add-ons, and shows the sources each answer cites. It does not track Claude. Its Insights, Actions and Impact loop connects the visibility change to GA4, Search Console and Cloudflare data, and its MCP server lets you query that data from Claude, ChatGPT or Cursor.
Fixing the files is step one; producing pages worth citing is step two. The guide to AI SEO content automation covers that pipeline.
AI-readiness checklist
- robots.txt allows OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot on the pages you want cited.
- Training bots are blocked only if you have decided you want that.
- llms.txt loads at the domain root, lists 10 to 25 pages and has a specific description for each.
- llms-full.txt exists only if you have documentation worth loading whole, and is regenerated on publish.
- Every page in the files returns HTTP 200 and renders its main content without JavaScript.
- Brand name, pricing, locations and product names match across your files, site and profiles.
- You have a fixed prompt set and a baseline of mentions and citations before any change.
Verdict
Publish llms.txt because it costs almost nothing and clarifies your own site architecture, but get robots.txt right first and put your real effort into crawlable, answer-ready pages and third-party sources. ai.txt is optional, and no file replaces content that engines want to cite.
To see whether your changes move the needle, start free on Listable Labs with no credit card, or ask for a free discovery audit if you want the team to do it for you.