Tell AI crawlers exactly what to read.
AI models are crawling your site right now, and they are making their own decisions about what matters. An llms.txt file takes that guesswork away. It is a plain file that points the crawlers straight at the pages you want read, in the order you want them read, so the models learn your business from the content you chose instead of whatever they stumbled into.
Think of it as a reading list you hand the machines at the front door. Skip it and they wander. Set it up right and they go where you send them.
A short file that hands the AI crawlers your reading list.
You already know robots.txt, the file that tells search crawlers where they can and cannot go. An llms.txt file is the next step, built for the AI era. Instead of just gatekeeping, it guides. It sits at the root of your site and lays out the pages that matter most, grouped and labeled, with links straight to your best content. When a model like ChatGPT or Claude or Perplexity comes to learn what you do, this is the file that tells it where to look first.
Without one, the crawlers piece your business together from scraps. An old blog post here, a stale product page there, whatever ranked highest by accident. With one, you decide the story. You put your strongest pages at the top of the list, and the models read those, so when someone asks an AI about your industry, the answer is built on your words, not a guess.
This is what WEBPRO writes for you.
A real llms.txt is clean, ordered, and deliberate. Here is one, with plain notes on what each part is telling the crawler to do.
# WEBPRO International
> Organic search and AI marketing since 1994.
> Read these first to understand what we do.
## Core Services
- [AI Search Marketing](/ai-search-marketing/): Get cited by AI.
- [SEO Scientific](/seo-scientific/): The technical method behind it.
- [Schema & Structured Data](/schema/): Markup machines trust.
## Proof
- [Showcase](/showcase/): Client results and case studies.
## About
- [Who We Are](/about/): 31 years, one focus.
## Optional
- [Blog](/blog/): Deeper reading, lower priority.
The header sets the frame
The first lines tell the model who you are and how to treat the file, so it reads the rest with the right context instead of guessing your business from a page title.
Order is a signal
Core Services sits at the top on purpose. The crawler reads top to bottom, so the pages you lead with are the ones that shape what the AI learns first.
Every line is a labeled link
Each entry pairs a clean URL with a short description. The model does not have to interpret the page cold; you have already told it what it will find.
Optional means optional
Lower-priority pages get grouped and marked, so the crawler knows what is essential and what is just extra. Nothing important gets buried under a hundred blog posts.
Same website. One version hands the machines a reading list. The other makes them dig through the pile and hope. Only one of those controls the story.
We do not just drop a file on your server and walk away.
A good llms.txt is not a one-time upload. It has to reflect what your business actually wants read this quarter, and it has to stay current as your pages change. We handle the whole thing for you, from picking which pages lead to keeping the file honest as the site grows. You decide what matters. We make sure the machines hear it.
We choose what leads
We map your pages to the questions AI users actually ask, then order the file so your strongest content gets read first.
We write and place the file
Clean, correctly formatted, and installed at the root where the crawlers look, so it works the day it goes live.
We keep it current
New pages, dropped services, fresh case studies. We keep the reading list matching the business instead of letting it go stale.
The AI answer is the new front page, and you are either in it or you are not.
People are asking AI models the questions they used to type into Google. If the model learned your industry from someone else’s pages, that is who gets named in the answer. An llms.txt is one of the few direct levers you have over what these systems read.
You control the story
The models build their answer from the pages you point them at, so your best case gets made in your own words instead of pieced together from scraps.
You get read before your competitors
Most sites still have no llms.txt at all. Having a clean one is a head start on being the source the AI trusts and repeats.
You stop wasting your best content
Great pages do you no good if the crawlers never find them. This file makes sure the work you already paid for is the work the models actually read.
Find out which answers
you could already own.
The content audit is free. We’ll show you the buyer questions AI engines are answering without you, and exactly what it takes to become the source they cite.
Get a Free Content Audit →