How to Configure robots.txt for AI Crawlers (2026)

TL;DR: Your robots.txt tells crawlers what they can read, and some of those crawlers now feed AI answers like ChatGPT, Perplexity, and Google's. Block the wrong ones and AI tools may skip your pages or describe you from stale sources. The fix is not to allow or block everything. AI crawlers come in four types; decide what each should see, and audit a file most teams set once and forget.

robots.txt is one of the oldest files on the web. For years it had one main reader: Googlebot. You set it once, usually when the site was built, and moved on.

That is changing. A growing share of buyer research now runs through AI assistants. Many of those assistants read robots.txt before they read your pages. The same small file now shapes two things at once: how search engines crawl you, and how AI systems access you.

This guide explains what robots.txt does for AI, the four kinds of AI crawler, and how to set the file up on purpose instead of by default.

Why this is worth reviewing now

A quick sense of scale, with the numbers stated plainly:

• Google's AI Overviews had over 1.5 billion monthly users, as of the company's Q1 2025 earnings call in April 2025.
• On that same call, Google said Search usage keeps growing nearly a year after the US launch, as people find it useful for more of their queries.
• 58.5% of US Google searches ended without a click to the open web in 2024, according to SparkToro and Datos.

Read these as context, not as proof of revenue impact. "Usage" is not the same as traffic to your site. A zero-click search is not always a lost opportunity. The useful takeaway is directional: more research now happens inside AI answers, so it is worth knowing whether those systems can read you.

What robots.txt does, and what it doesn't

robots.txt is a plain text file at the root of your domain, at yourdomain.com/robots.txt. It gives instructions to automated crawlers using three directives:

• User-agent: which crawler the rule applies to.
• Disallow: paths the crawler should not access.
• Allow: paths it can access.

Anyone can read yours right now, including competitors. Two limits are important before you change anything.

It is voluntary. Well-behaved crawlers from major AI companies respect robots.txt. Some crawlers ignore it, and enforcement varies by company. Treat the file as a request, not a lock.

It controls crawling, not memory. Blocking a crawler today does not erase what a model already learned during training. It does not remove content that lives on other sites. Blocking a training crawler does not make an assistant forget you. It changes whether that system can read your site fresh, going forward.

Here is why that matters in practice. When an AI system cannot read your site, it does not go silent. It fills the gap with other sources: reviews, directories, cached pages, marketplace listings, and comparison content. Sometimes that is accurate. Sometimes it is out of date, or written by someone else.

The four kinds of AI crawler

The common mistake is treating "AI bots" as a single thing to allow or block. They are not. They fall into four types, and each one does a different job.

A few points to hold onto.

Training crawlers are a reasonable thing to block if training is your concern. That choice is mostly separate from whether you appear in AI answers today.

User-triggered fetchers are the type most worth keeping open. When someone asks an assistant to "compare these three tools," this is the crawler that opens your pages to do it.

Google-Extended is a special case. It is not a normal crawler. It is a control that tells Google whether your content can be used for Gemini and Vertex AI training. Blocking it does not remove you from Google AI Overviews. Those run on Google Search systems, which depend on Googlebot access and snippet controls like data-nosnippet and max-snippet, not on the AI-training toggle.

Takeaway: The four types are stable. The exact crawler names change often. Verify the current user-agent strings against each provider's documentation before you finalize rules. The names in this guide are examples as of July 2026.

How blocking a crawler changes what an assistant can see

A short walk-through of the mechanism. Say a buyer asks an assistant: "best onboarding tool for enterprise SaaS, with transparent pricing." The assistant tries to open the top candidates' pricing and feature pages to compare them.

If your site allows those fetchers, your pages go into the comparison in your own words. If it blocks them, the assistant does one of three things:

• Leaves you out, because it cannot verify your claims.
• Notes that your pricing or details are unavailable.
• Uses an older third-party description instead.

For a concrete look at how these AI shortlists come together, see our breakdown of the CRMs ChatGPT recommends and where the answer comes from.

Two honest caveats keep this accurate. Allowing a crawler does not guarantee you get cited. Blocking one does not guarantee you disappear, because the model may already know you from training or other sources. Access is a precondition, not a promise. It puts you in a position to be read.

For technical products, the pages that matter most in an AI comparison are usually the same ones humans evaluate:

• Docs and API references
• Integration pages
• Security and compliance pages
• Pricing pages
• Changelogs and release notes

If assistants cannot reach those, a capable product can look thin in a summary, simply because the evidence was not readable.

Which sites this matters most for

Three situations to recognize.

Documentation-heavy software is affected first. Much of the evaluation lives in pages an assistant needs to read.

Local and ecommerce sites are affected in a different way. If your pages are blocked but competitors are open, assistants lean on directories, maps, and review sites. For this group, structured data, Google Business Profile, and product feeds often matter as much as robots.txt.

Then there is accidental over-blocking. A team reads about AI training concerns and adds a blanket block, sometimes through a firewall rule that stops every crawler. The intent is reasonable. The result is too broad, because it treats a training question as a reason to close off search and user-triggered access as well. The better approach is to separate the four types.

How to decide what to allow or block

A simple default that works for most B2B sites:

• Keep traditional search open (Googlebot, Bingbot). Blocking these has large, well-known costs.
• Keep search/indexing and user-triggered fetchers open on public pages, so assistants can read and cite you.
• Decide on training crawlers deliberately. Blocking them is a valid privacy or policy choice, and it does not remove you from AI answers.

If you are unsure, the safe starting position is to keep everything open except the training crawlers you have a specific reason to block.

A robots.txt starting point you can copy

Here is a balanced default that reflects the decisions above. Adjust the training block to match your own policy.

# Normal search indexing: keep open
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI search + user-triggered assistants: keep open so they can read and cite you
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

# Model-training crawlers: your policy decision.
# Blocking these does NOT remove you from AI search answers.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everything else: default allow for public pages
User-agent: *
Allow: /

Do not paste this without checking three things:

• robots.txt allows what it does not forbid by default. Several Allow lines above just make intent clear. They do not unlock anything on their own.
• Group order and user-agent spelling matter. A misspelled bot name silently does nothing.
• A rule works only if the crawler honors it and nothing upstream overrides it. A CDN or firewall filter can block agents no matter what this file says.

How to audit your robots.txt in 20 minutes

You can find most problems in one sitting. Work through these steps in order:

1. Read the main file. Open yourdomain.com/robots.txt line by line. Look for a site-wide Disallow: /, and for named AI agents under Disallow rules you did not intend.
2. Check your subdomains. robots.txt is host-specific. docs.example.com, app.example.com, and help.example.com each have their own file. Teams often check www and miss the docs subdomain, where the real evaluation content lives.
3. Check the CDN or firewall layer. A "block AI bots" setting there can filter the same crawlers, and robots.txt will not reveal it.
4. Test the outcome. Ask two assistants a real buyer question in your category. Note whether you appear, how you are described, and which sources they cite.

If you are missing or described from stale sources, your access rules are one of the first things to check.

Common mistakes to avoid

• Blocking every AI crawler "for safety." This gives up search and user-triggered visibility to solve a training concern.
• Allowing everything without a decision. Training access is a separate, legitimate choice worth making on purpose.
• Assuming one robots.txt change guarantees AI recommendations. Being readable gets you into the running. Being useful, well-reviewed, and cited across the third-party sources AI engines trust is what earns a recommendation.
• Editing only the www file and forgetting subdomains and the CDN layer.

How to check whether your change worked

Editing robots.txt removes a blocker. It does not tell you whether the blocker cost you anything, or what shifted afterward. To see that, you need to look at how AI systems actually describe and rank your brand.

That is the job of an AI visibility tool. The robots.txt fix is a first step; improving brand visibility in ChatGPT is the larger project around content, citations, and sources. SupaIntent measures where your brand appears across ChatGPT, Gemini, Perplexity, and Google AI Mode, which competitors get recommended instead, and which sources shape those answers. It also generates an llms.txt file and flags content and technical gaps to close.

Related guides

• Keep sitemap.xml fresh for AI search
• Check whether JavaScript hides your product from AI crawlers

FAQ

How do I configure robots.txt for AI crawlers?

Group the crawlers by type. Keep traditional search (Googlebot, Bingbot), search/indexing agents (OAI-SearchBot), and user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) open on your public pages. Then make a deliberate choice about training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended). Verify current user-agent names against each provider's docs before you publish.

Should I block AI crawlers in robots.txt?

Only the ones you have a specific reason to block, usually training crawlers, if training use is a concern. Blocking search and user-triggered crawlers reduces how well AI assistants can read and cite your pages, so most sites keep those open.

Does blocking GPTBot stop ChatGPT from mentioning my company?

No. GPTBot is a training crawler. Blocking it affects future training, not whether ChatGPT's live search features can retrieve your current pages. Those use different agents like OAI-SearchBot and ChatGPT-User. You can block training and stay visible in AI search.

Does blocking AI crawlers affect Google AI Overviews?

Not through the AI-bot rules. AI Overviews run on Google Search systems. They depend on Googlebot access, indexing, and snippet controls like data-nosnippet and max-snippet, not on whether you block third-party AI crawlers or Google-Extended.

How do I know if my site is currently blocking AI crawlers?

Read yourdomain.com/robots.txt for Disallow rules on named AI agents or a site-wide block. Check each subdomain separately. Then check your CDN or firewall for an "AI bot" filter. Confirm by asking a few assistants a buyer question in your category and seeing whether you appear.

robots.txt is a small file, and reviewing it takes less time than most people expect. The next time an AI assistant answers a question in your category, it will read what your site allows and infer the rest. Knowing what it is allowed to read is a reasonable thing to check once, and then again whenever your site structure changes.