Three ways a site can block AI
- robots.txt rules. We read your robots.txt the way the crawlers do (following the official RFC 9309 matching rules) and work out, bot by bot, whether your page is allowed.
- Your firewall. We request your page as each AI crawler and as a normal browser. If people get in but the bot gets a 403 or a challenge page, a firewall or security plugin is blocking it, even if robots.txt says yes.
- Page-level tags. We check meta robots and the X-Robots-Tag header for noindex and noai, which can stop your content being used even when crawlers get in.
AI search bots vs AI training bots
Not every AI crawler does the same job, and blocking the wrong ones is the most common mistake we see.
| Type | What it does | If you block it |
|---|---|---|
| AI search | Fetches pages to answer questions in ChatGPT search, Claude, Perplexity, Google AI Overviews and Copilot, usually with a link back to you. | You disappear from AI answers and lose that referral traffic. |
| AI training | Collects content to train future AI models, for example GPTBot, ClaudeBot, Google-Extended and CCBot. | Your content isn't used for training. This does not remove you from AI search answers. |
For most Australian businesses the sensible default is: allow AI search bots, and make a deliberate choice about training bots. Blocking training is reasonable if you publish original content you don't want reused. Blocking search bots almost never is.
Why your site might block AI without you knowing
- Cloudflare's AI bot blocking. Cloudflare lets sites block AI crawlers with a single setting, and offers it when new sites are set up. It is easy to switch on without realising it also blocks AI search, not just training.
- Security plugins and hosting firewalls. WordPress security plugins and some hosts block unfamiliar bots by default.
- Copied robots.txt templates. Lists of "bad bots" shared online often include AI search crawlers alongside genuine scrapers.
- A broken robots.txt. If robots.txt returns a server error, well-behaved crawlers assume the whole site is off limits.
How to allow AI search bots in robots.txt
Add a group for each AI search crawler you want to welcome. More specific groups override the general User-agent: * rules, so these bots get in even if your default is strict.
User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / # Optional: opt out of AI training only User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: /
Then run the checker again. If a bot still shows as blocked by your firewall, the fix is in your security settings rather than robots.txt.
What this checker can't tell you
Our live test sends requests with each crawler's name, from our own servers. A block for that name is strong evidence your firewall is turning the bot away. A pass means you aren't blocking by name, but some firewalls also verify a crawler's real network address, so the genuine bot could still be treated differently.
Being accessible is also only the first step. Whether AI tools actually recommend you depends on your content, reviews and how clearly your pages answer real questions. That's where our team comes in.