AI Crawler Control: What to Allow, What to Block, and Why

The Two Word Question That Costs Brands Their Citations

“Block bots.” Two words, usually said in a security review, usually acted on within the hour. Six months later somebody notices the brand has quietly disappeared from AI answers and nobody can work out why.

This happens because “AI crawler” is not one thing. Some crawlers collect content to train future models. Some fetch a page in real time so a model can answer a user’s question right now and cite the source. Blocking the first group is a legitimate business decision. Blocking the second group removes you from the answer. The robots.txt line looks nearly identical either way, which is exactly why this goes wrong so often.

AI crawler control is the practice of deciding, deliberately and per agent, who gets to read your content and for what purpose. At Sympler we treat it as a strategy decision that happens to be implemented in a text file, not an IT task that happens to have strategy consequences.

The Three Jobs AI Crawlers Do

Before touching a single directive, get clear on categories. Nearly every bad outcome traces back to collapsing these three into one.

• Training crawlers gather text to train or fine tune models. Allowing them may contribute to a model’s general knowledge of your brand. It sends you no traffic and gives you no citation.
• Retrieval crawlers maintain an index that an AI product searches when a user asks something. This is the category that gets you cited with a link.
• User triggered fetchers load a specific page because a person pasted a URL or the assistant decided to open it mid conversation. These are closer to a human visitor than a crawler.

The strategic default for most businesses is to allow retrieval and user triggered access while making a considered choice about training. That default sits inside the broader AI SEO strategy we recommend heading into 2026. Publishers with licensing leverage often decide differently, and that is a reasonable position. What is not reasonable is arriving at a blanket block by accident.

Who Is Actually Knocking

These are the agent tokens worth knowing. Names and behavior do change, so confirm against each operator’s current documentation before you deploy anything.

CCBot deserves a note. Common Crawl is a public dataset that many organizations build on. Allowing it means your content may end up in training corpora you never evaluated individually. That is a one to many decision worth making consciously.

The Google-Extended Trap

This one causes more damage than any other single misunderstanding, so it is worth stating plainly.

Blocking Google-Extended does not remove you from Google Search, and it does not remove you from AI Overviews. AI Overviews are generated from Google’s search index, and that index is governed by Googlebot. Google-Extended is a separate control covering the use of your content for Gemini and Vertex AI model training.

The practical consequence is that a team wanting to stay out of AI Overviews cannot achieve it with Google-Extended. The only lever that reliably removes you from Google’s AI surfaces is also the lever that removes you from Google Search, which almost no commercial site wants to pull. Anyone who tells you there is a clean way to keep your organic rankings while opting out of AI Overviews is overselling. If that tradeoff is central to your planning, our GEO and SEO playbook works through the strategic side of it.

What llms.txt Is, and What It Is Not

You have probably been asked about llms.txt. Here is a straight answer.

It is a proposed convention, introduced in late 2024, for placing a markdown file at the root of your domain that points AI systems to your most important, cleanest content. The intent is genuinely sensible: give a model a curated map instead of making it infer structure from a sprawling site.

Now the part vendors tend to skip. As of early 2026, no major AI provider has publicly confirmed that llms.txt influences whether or how their systems retrieve or cite your content. It is a community proposal with real momentum and no confirmed enforcement. Treat any claim that it “gets you into ChatGPT” as unsupported.

So should you publish one? Probably yes, with clear expectations:

• It costs very little to create and maintain.
• It is a useful forcing function, because deciding which twenty pages represent your business is valuable work regardless of who reads the file.
• It is not a substitute for the things that demonstrably matter, which are crawl access, clean structure, and consistent entity signals.

Publish it, keep it accurate, and do not build a strategy on it. If it becomes a supported standard you are already positioned. If it does not, you spent an hour.

What a Useful llms.txt Contains

Keep it to a short title, a sentence or two describing what the organization does, and a curated list of links with brief descriptions. Your best explanatory pages, your core service pages, your documentation. Not every URL you have. A dump of your sitemap defeats the entire purpose of curation.

Robots.txt Is a Request, Not a Lock

An important limitation that gets glossed over: robots.txt is a voluntary protocol. Well behaved crawlers respect it. Poorly behaved ones ignore it entirely, and some spoof user agent strings to look like browsers.

If your requirement is genuinely “this content must not be scraped,” robots.txt does not deliver that. Enforcement lives at the infrastructure layer through your CDN or web application firewall, using verified reverse DNS or published IP ranges rather than the self reported user agent string. Most enterprise CDNs now ship managed rules for known AI agents.

Match the tool to the requirement:

• Signalling a preference to cooperative crawlers is a robots.txt job.
• Preventing access is an infrastructure job.
• Protecting genuinely sensitive material is an authentication job. Anything valuable enough to protect properly should sit behind a login, not behind a polite request.

For organizations running large or multi property estates, this coordination problem is usually the hard part. Our enterprise SEO practice spends a surprising amount of time reconciling directives that three different teams deployed without talking to each other.

A Decision Framework You Can Actually Use

Work through these in order. Do not skip to the file.

1. Establish your position on training. Is your content a differentiated asset with licensing value, or is it marketing material whose job is to be seen? Publishers and marketers usually land in different places, and both are correct for their business.
2. Confirm you want retrieval access. For almost every commercial site the answer is yes. Being cited in an AI answer is earned media.
3. Audit what you have deployed today. Read your live robots.txt, your CDN rules, and any plugin that writes directives. Conflicts between these three are extremely common.
4. Write directives per agent. Name specific tokens. Avoid broad wildcard disallows unless you have deliberately decided to exit AI surfaces entirely.
5. Verify from outside. Check your live file from a browser, not from your staging environment. Then check server logs to confirm the agents you expect are actually arriving.
6. Review quarterly. New agents appear regularly. A file written eighteen months ago is describing a landscape that no longer exists.

The Mistakes We See Most

• Blocking everything after a scraping incident. An understandable reaction that often costs more in lost visibility than the scraping cost in the first place.
• Blocking retrieval crawlers while expecting citations. These are mutually exclusive. You cannot be quoted from a page nobody is allowed to read.
• Assuming Google-Extended controls AI Overviews. Covered above, and worth re-checking on your own site today.
• Leaving a staging disallow in production. Still, after all these years, one of the most expensive two line mistakes in the business.
• Treating robots.txt as security. Publishing a disallow list is publishing a map of what you consider sensitive.

Frequently Asked Questions

Will blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended does not affect your organic rankings, because it is a training control and not an indexing control. Blocking Googlebot absolutely does affect rankings, and that is the directive to handle with care. Third party AI crawlers such as GPTBot or PerplexityBot have no bearing on Google rankings at all.

Do I need both robots.txt and llms.txt?

They do different jobs and do not overlap. Robots.txt states who may crawl and is a widely respected standard. Llms.txt suggests what is worth reading and is an unconfirmed proposal. Robots.txt is the one that currently has effect.

How do I tell whether AI crawlers are visiting?

Server access logs are the source of truth. Filter by user agent for the tokens above and look at request volume and which paths are being fetched. Analytics platforms generally will not show this, because most crawlers do not execute the JavaScript that analytics depends on.

Can I allow AI systems to cite me but not train on my content?

Partially, and only where an operator publishes separate tokens for those purposes. OpenAI, Google, and Apple currently offer some separation. Others do not, which leaves you with an all or nothing choice for those agents. Anyone describing this as fully solved is oversimplifying.

Does any of this apply to a small local business?

The training question matters less when your content is service pages rather than a publishing archive. The retrieval question matters enormously, because AI answers to local queries are short and being excluded is close to being invisible. Check that you are not accidentally blocking retrieval, then move on to more valuable work.

Get Your Current Configuration Checked

Most sites we audit have at least one directive nobody on the current team wrote and nobody can explain. Sometimes it is harmless. Occasionally it is the reason a brand has been missing from AI answers for a year.

It takes very little time to find out which situation you are in. Read your live robots.txt today, then reconcile it against your CDN rules. If you would rather have someone experienced look at the whole picture, including how your website architecture and AI SEO setup interact with these directives, request a free analysis or contact our team and we will walk your configuration with you line by line.

Sympler Web Team