Your robots.txt Is Ignoring Half Your Rules: What Actually Works for AI Bots

Somewhere right now, a website has a line in its robots.txt that says No-AI: please. It feels very official. It does absolutely nothing.

Google's own Search Central team explained exactly what robots.txt understands. Spoiler: it's a very short list.

Let's fix your file before it embarrasses you any further.

Quick words (one line each):

• robots.txt: a small text file at yoursite.com/robots.txt that tells bots where they can and can't go.

• Crawler (bot): a program that visits pages to read them, like Googlebot.

• User-agent: the name a bot uses to introduce itself (like GPTBot).

• Training bot: a bot collecting content to train an AI model.

• Search bot: a bot collecting pages so an AI tool can show and cite them in answers.

Step 1: Learn the only four rules that count

Google Search Central says robots.txt follows a standard (RFC 9309) and well-behaved crawlers read it first (1:03, Oct 21, 2026). Then the key part: only four rules count (3:02-3:32):

1. user-agent (which bot this group is for)

2. allow (where it can go)

3. disallow (where it can't go)

4. sitemap (where your sitemap lives)

Lines starting with # are comments, for humans only (2:31).

Anything else, like "don't crawl" or "no-ai," is dropped silently, with no error. Your file looks strict. The bots see a blank space.

Step 2: Delete the made-up lines (they're decoration)

Open yoursite.com/robots.txt and read every line. If a line doesn't start with one of the four rules or a #, it's decoration.

Google says custom rules like "no-ai" are ignored (7:06-8:00). Remove them, or turn them into a real rule (next step). Charlie Whitworth says cleaning up robots.txt (along with pruning weak pages) lifted AI referral traffic almost right away for his clients (36:00-37:30, Oct 4, 2026).

Step 3: Name the bot, not the vibe

Google says to block a certain AI crawler, name its own user-agent token (7:06-8:00). One group per bot.

Know the difference between search and training bots. Victorious says OAI-SearchBot is the one ChatGPT search uses, so allow it if you want to show up in ChatGPT (13:44-14:13, Oct 1, 2026). GPTBot is OpenAI's training bot. They're separate.

Example: allow everything, but keep OpenAI's training bot out of one folder:

# Everyone is welcome
User-agent: *
Allow: /

# Keep OpenAI's training bot out of the research folder only
User-agent: GPTBot
Disallow: /research/

Sitemap: https://yoursite.com/sitemap.xml

Bot names change. Check each company's own docs for the current token before you edit.

Step 4: Block narrowly, if you block at all

Victorious says make crawler choices on purpose. If you must block training bots, block one folder, not the whole site. Keep your home page, services, about and case studies open, and keep search bots allowed everywhere you want to be found (28:36-29:10).

Step 5: Don't count on Google-Extended

Victorious says blocking Google-Extended does NOT keep you out of Google AI Overviews (11:15-12:08). If your plan was "block Google-Extended and disappear from AI answers," that plan has a hole in it.

He adds that old nosnippet and max-snippet tags DO limit what shows in Google's AI features. Search your page source for them.

Step 6: Keep the file clean and boring

Two formatting rules from Google Search Central:

1. User-agent names use only letters, dashes and underscores. No emoji, no fancy symbols (4:04). That matches RFC 9309, the robots.txt standard, which allows only letters, hyphens and underscores in a bot's name.

2. Non-English paths must be percent-encoded (special characters written as codes like %C3%A9) so every crawler reads them the same (5:33-6:30).

Step 7: Remember what robots.txt can't do

robots.txt is a "please" sign, not a lock.

• Google says it controls access, not use. It decides whether a bot is allowed to visit, not what the company does with your content afterward (7:06-8:00).

• Victorious says it's only a request. If legal needs a real block, do it at the server or CDN. He cites a vendor test where about 4 in 10 sites that banned GPTBot still served it pages (29:17-30:55).

Step 8: Check your CDN and logs

A CDN (content delivery network, like Cloudflare) sits in front of your site and can block bots before robots.txt even matters.

Charlie Whitworth warns that a "block AI training bots" type setting can also end up blocking Googlebot, Applebot and Bingbot (58:00-59:30). Then use your log files to see which bots crawl which pages, and watch for fake "AdsBot" visits (1:01:30-1:04:31).

Logs are the receipt. Everything else is a guess, and guessing is how you block Googlebot by accident.

Do this today

1. Open yoursite.com/robots.txt.

2. Delete any line that isn't user-agent, allow, disallow, sitemap or a # comment.

3. Make sure OAI-SearchBot, Googlebot and Bingbot are allowed on pages you want found.

4. If you block a training bot, name it and block one folder, not the whole site.

5. If you use Cloudflare, check its bot settings today.

Related reading

• Technical SEO for AI Search: What to Keep, Start and Stop Doing

• The 7-step bot access check

Sources

• Google Search Central ? What can you actually put in a robots.txt? (2026-10-21)

• Victorious (The Search Signal) ? Technical SEO for AI Search - What to Keep, Start, and Stop Doing (2026-10-01)

• Candour Agency with Charlie Whitworth ? SEO vs GEO - Which Traditional SEO Tactics Still Work for AI Search (2026-10-04)

Don't worry if this seems like it's a pain in the ass for someone who doesn't build websites. That's because it is. The good news: just click the link below, pull the trigger, and we'll take care of the rest.