Is Your Site Locked Out of AI Search? The 7-Step Bot Access Check
Technical SEO
Step-by-step
AI crawler access check
Trigger Marketing
The Trigger team
⊹
You can write the best page on the internet. If the bot can't get through the door, it is a lovely page for nobody.
So before you rewrite anything, check that the robots that power AI answers can actually read your site. Kasra Dash makes "can it be crawled" one of the first items on his list. Victorious walks through the same check. This post turns both into seven steps you can run on any site.
Quick words (one line each):
Bot / crawler: a program that visits web pages and reads them.
robots.txt: a plain text file at yoursite.com/robots.txt that asks bots to stay out of certain places. It is a request, not a lock.
CDN: a service that sits in front of your site (Cloudflare is one) to make it fast and filter traffic.
Server log: the diary your server keeps of every visit.
Step 1: Open your robots.txt
Type yoursite.com/robots.txt into a browser. Read it.
Make sure Googlebot, Bingbot and OAI-SearchBot are allowed on the pages you want found. OAI-SearchBot is the bot behind ChatGPT search. GPTBot is a different bot, used for training. They are separate, so one rule does not cover both (Victorious, Oct 1, 2026).
Step 2: Check any "block AI" rules for collateral damage
Some sites block AI training on purpose. That is fine. Block it narrowly: a folder, not the whole site, and keep the search bots open (Victorious).
One myth to drop: blocking Google-Extended does not remove you from Google AI Overviews. It has no effect on them (Victorious). And Google's AI Overviews and AI Mode use the normal Google index, so if Google can index the page, that part is covered.
Step 3: Check your CDN and firewall
Your robots.txt can say "come in" while your CDN slams the door.
If your site is on Cloudflare, look at its bot settings. Victorious says Cloudflare's free-plan defaults, from around Sept 15, block training and agent crawlers on pages that show ads. Kasra Dash lists the Cloudflare check right next to "is the site crawlable" as one of the first items.
Look for the setting. Don't assume it is off.
Step 4: Prove it with logs
robots.txt is only a request. Bots can ignore it. Victorious cites a vendor test where about 4 in 10 sites that banned GPTBot still served it pages.
So don't trust the file. Open your server logs or your CDN's bot report. Confirm the search bots get normal, successful responses on your key pages.
Step 5: Hunt for old snippet limits
Search your page source for nosnippet and max-snippet. These old tags limit how much of your page Google can show, and Victorious says they limit Google's AI features too.
Step 6: Turn JavaScript off and look at the page
JavaScript is the code that builds parts of a page in your browser. Victorious says most AI crawlers do not run it. If your company name or description only appears after JavaScript runs, the bot sees an empty page.
Turn JavaScript off in your browser tools and reload. You should still see:
your company name
a plain description of what you do
key facts such as prices, hours or service areas
Victorious also says to treat every page as standalone: company name and a short description near the top of each page.
Step 7: Check Bing and Brave
Victorious says Copilot uses Bing and Claude uses Brave. So:
Set up Bing Webmaster Tools and submit your sitemap.
Search 3 to 5 questions a buyer would ask in Brave Search. See whether your site shows up.
Write down the scorecard
One line per check: pass, fail or not sure, then the fix and who does it. That is the whole deliverable.
Heads-up before you touch anything
Bot names change. Confirm the current names in each company's own docs before you edit a robots.txt.
Ask the site owner before changing robots.txt or CDN settings. This is how people accidentally block Google at 4 PM on a Friday.
Both sources sell SEO tools or services, and the 4-in-10 number is a vendor test, not an independent study. The checks themselves cost nothing, so run them anyway.
This check proves bots can read you. It does not prove an AI will quote you. That is a different job.
Do this today
Open yoursite.com/robots.txt and confirm Googlebot, Bingbot and OAI-SearchBot are allowed.
Check your CDN's bot settings.
Reload your top page with JavaScript off. Is your company name there?
Write pass / fail / not sure next to each of the 7 steps.
Related reading
Sources
Don't worry if this seems like it's a pain in the ass for someone who doesn't build websites. That's because it is. The good news: just click the link below, pull the trigger, and we'll take care of the rest.
Share This Article
