llms.txt vs robots.txt: GPTBot, ClaudeBot, and PerplexityBot
llms.txt vs robots.txt is not a ranking contest: robots.txt tells crawlers what they may fetch, while llms.txt is an optional markdown file that describes the site for agents. Neither one puts you in ChatGPT.
What is the difference between llms.txt and robots.txt?
robots.txt is an access file. It lives at `/robots.txt` and uses `User-agent`, `Allow`, and `Disallow`. Reputable crawlers treat it as the site’s stated rules. It answers one question: may this bot fetch this path?
llms.txt is a content file. The informal spec at llmstxt.org asks for a markdown document, usually at `/llms.txt`, that starts with an H1 for the site name, then a short summary and curated links. It has no Allow and no Disallow. It cannot block GPTBot. It cannot unblock a path you already closed.
Think of them as two jobs on the same host. robots.txt is the door policy. llms.txt is a short map you leave out for an agent that is already allowed in.
A sitemap lists URLs for discovery. It is not a substitute for either file. It will not tell a model what matters, and it will not stop a crawler you meant to refuse.
What do GPTBot, ClaudeBot, and PerplexityBot actually do?
The names in the title are not one job shared three ways. Each vendor splits training, search, and user-initiated fetches. A rule for one token does not apply to the others.
OpenAI documents GPTBot as a training crawler. Disallowing it is a signal that future content should not be used to train their foundation models. ChatGPT search is a different token: OAI-SearchBot. You can allow search and disallow training. ChatGPT-User is used when a person asks ChatGPT to open a page. OpenAI notes that user-initiated fetches may not follow robots.txt the same way.
Anthropic documents ClaudeBot as training, Claude-SearchBot for search quality, and Claude-User for a fetch a person asked Claude to make. Anthropic says those bots honour robots.txt. Blocking ClaudeBot does not block Claude-SearchBot.
Perplexity documents PerplexityBot as the crawler that surfaces sites in Perplexity search, and says it is not used to train foundation models. Perplexity-User is the fetch that happens when someone asks a question. Perplexity says that fetcher generally ignores robots.txt because a person requested the page.
So “block the AI bots” is not a single line. If you Disallow GPTBot and leave OAI-SearchBot open, you have stated a training preference, not a ChatGPT-search preference. If you Disallow PerplexityBot, you are talking to their search crawler.
None of this is a promise that a model will cite you. robots.txt is a request. The file is still the place you write the policy.
Does llms.txt replace robots.txt?
No.
llms.txt does not grant access, deny access, or target a user-agent. If GPTBot, ClaudeBot, or PerplexityBot is disallowed in robots.txt, a markdown file at `/llms.txt` will not override that.
The spec describes llms.txt as something an agent reads on demand: a short index, then links to cleaner pages when it needs more. That is useful for a SaaS that wants an agent to see the product, the API, and the pricing page without wading through a full HTML shell. It is not a lever for crawl volume.
Write the door policy first. Then, if you want a curated path through the pages a bot is allowed to fetch, add llms.txt.
What can you measure on a live URL?
You can look. You cannot invent a visibility score.
On a live SaaS URL it is fair to ask: is there an llms.txt? Is there a robots.txt? Which user-agent rules are written for GPTBot, ClaudeBot, PerplexityBot, and the search or user tokens beside them? Are those rules labelled so you can see the source?
IvanHub can measure labelled AI-search checks on a live URL. The checks are labelled and experimental. They are not a ranking predictor. They do not mean ChatGPT will mention you. They mean the files and rules on that host are visible, with evidence attached. If a check is unavailable, do not invent the result.
How should a SaaS treat both files?
Decide the policy before you write markdown.
If you do not want training crawls, say so on GPTBot and ClaudeBot. If you want to be eligible for a vendor’s search crawl, say so on that search token. Do not assume a blanket `User-agent: *` line did what you meant.
Then write an llms.txt a person could read in a minute: who you are, what the product does, and links to the pages that matter. Point at pages that are allowed. A map to a disallowed path is noise.
What we measure
- Labelled AI-search checks on a live URL
- Presence of llms.txt, and parsed contents of robots.txt, when they exist
- Evidence on the finding, not an invented AI-visibility score
These checks are experimental. They do not rank you in ChatGPT.