AI Crawler Access: The robots.txt and WAF Audit That Comes Before Any Content

Short answer, as of August 2026: if an AI engine never cites you in your industry, check crawler access before you check anything else — and check it at the firewall, not just in robots.txt. Allowing the retrieval agents costs nothing, takes about ten minutes, and is a hard prerequisite: an engine that cannot fetch your page cannot quote it, no matter how good the page is.

Here is the full audit, in the order worth running it.

Two different gates, and only one of them is polite

There are two independent places your site can refuse an AI crawler, and they fail in different ways.

robots.txt is a request. It's a text file the crawler reads and voluntarily honours. The major AI companies do honour it. If it disallows their agent, they stop — silently, with no error anyone on your side ever sees.

Your WAF or CDN is an enforcement layer. Cloudflare, AWS WAF, Akamai, Sucuri and most managed hosts ship bot-management rules that block or challenge unfamiliar user-agents by default. This gate returns a 403, a 503, or a JavaScript challenge page. The crawler gets something — just not your content.

The second gate causes far more silent failures than the first, because nobody edited anything to turn it on. It arrived with the plan. Cloudflare in particular began offering one-click AI-crawler blocking to all customers in 2024 and later moved toward blocking AI training crawlers by default on new zones — meaning a site launched recently may be blocking these agents without anyone ever having made that decision.

The agents that matter, and what each one does

Not every AI user-agent does the same job, and the distinction matters: some fetch pages to train models, others fetch pages to answer a question right now. Blocking the first has no effect on citations. Blocking the second removes you from the answer.

The pattern is worth internalising: the "-User" and "-SearchBot" style agents are the retrieval path. The plain training crawlers are a separate, genuinely optional policy question — you can refuse to be training data and still be cited constantly. Which engines lean on which of these indexes is covered in more detail in how ChatGPT, Perplexity and Gemini pick their sources.

A robots.txt that allows retrieval

The minimum viable configuration allows the retrieval agents explicitly, before any restrictive catch-all:

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Three implementation notes that catch people out:

• Records match on the most specific matching user-agent group only. A crawler that matches its own named group ignores the User-agent: * group entirely. So a named group with an incomplete rule set doesn't inherit the general one.
• Framework-generated robots.txt can be shadowed by a static file. Next.js apps often have both an app/robots.ts route and a public/robots.txt. Which one wins is not intuitive and has flipped between versions. Curl the live URL and look at the actual bytes — never assume from the source tree.
• A blanket disallow on /wp-content/ or /assets/ can break rendering. If the crawler can't fetch the CSS and JS a page needs, what it sees may not be what you published.

The WAF check nobody runs

robots.txt being correct proves nothing about whether requests succeed. Test it the way the crawler experiences it — a plain HTTP request with the agent's user-agent string, from outside your network:

curl -A "OAI-SearchBot" -sSI https://example.com/your-best-page
curl -A "PerplexityBot" -sSI https://example.com/your-best-page
curl -A "ClaudeBot" -sSI https://example.com/your-best-page

You want HTTP/2 200. Treat anything else as a finding:

• 403 or 406 — a WAF rule or bot-management setting is blocking the agent by name.
• 503 with an HTML body — a JavaScript interstitial challenge. Crawlers do not solve these. Functionally identical to a block.
• 429 — rate limiting. Legitimate at scale, but if it fires on a single request your thresholds are too tight for crawlers.
• 200 but a tiny body — you're being served a challenge or a cached shell rather than the page. Drop the -I and read the actual HTML; if the content isn't in it, that's a rendering problem, covered in why client-side rendered content is invisible to AI engines.

Run the same three commands against a competitor who is being cited. When the diagnosis is "they answer 200 and we answer 403," the entire strategy question collapses into one firewall rule.

Where the block usually lives

In rough order of how often each turns out to be the culprit:

1. Cloudflare bot management — the AI Crawlers control panel, plus any "Block AI Scrapers and Crawlers" toggle. This is the single most common cause.
2. A managed WordPress host's default security profile — WP Engine, Kinsta and similar ship aggressive agent filtering that you don't administer from inside WordPress.
3. A security plugin — Wordfence, Sucuri and friends maintain their own bot lists that update themselves.
4. A hand-rolled nginx or Apache rule — often added years ago against a scraper, with a regex broad enough to catch everything ending in bot.
5. Geographic or ASN blocking — crawler traffic originates from cloud ranges. A rule blocking datacentre ASNs blocks every crawler along with the abuse it was aimed at.

If you want to keep a hard security posture, allow the agents by verified identity rather than by user-agent string alone. OpenAI, Anthropic, Perplexity and Google all publish IP ranges for their crawlers, and several now support HTTP message signatures so a request can be cryptographically attributed. Allow-listing on verified identity gives you the citation upside without opening a spoofable header to everyone.

What allowing them actually buys you

Be clear-eyed: access is necessary, not sufficient. Fixing a 403 does not produce citations on its own — it produces eligibility. What it removes is the failure mode where every other investment is wasted because the page was never readable.

The other four causes of not being cited — thin retrievability, weak entity signals, no third-party corroboration, and content that isn't quotable — are diagnosed in what to do when a competitor is cited in AI answers and you aren't. The content standard that makes a fetched page worth quoting is in how to get cited by ChatGPT and Perplexity, and the markup layer that makes the facts machine-readable is in structured data, llms.txt and AI citation.

Making it a monitor, not a one-off

The reason this belongs in software rather than a checklist is that access regresses. A CDN plan change, a plugin update, a new WAF managed ruleset, a migration — any of these can reinstate the block, and the symptom is silence. You find out months later when someone asks why the AI numbers flattened.

DigiRank's site audit and technical auto-fix pipeline re-checks crawler reachability on a schedule and flags a regression as an issue rather than leaving it to be noticed. It runs alongside the AI Visibility Tracker, so a drop in citations and the access change that caused it appear in the same timeline instead of being correlated by hand months later. Access monitoring is included from the $99/mo Starter plan and the audit-plus-auto-fix pipeline from Agency at $249/mo with a 14-day trial. It connects to Search Console, Bing Webmaster Tools and GA4 so a crawl regression shows up next to its traffic consequences.

Run the curl commands today. If they all return 200, you've spent ten minutes buying certainty about the one variable that makes everything downstream pointless when it's wrong.

Frequently asked questions

How can I get cited by ChatGPT in my industry? Start by confirming ChatGPT's retrieval agents can fetch your pages — request your top pages with the OAI-SearchBot and ChatGPT-User user-agents and confirm a 200 response. After access, the requirements are a page that answers the specific question directly, verifiable specifics such as figures and named exceptions, consistent entity information across the web, and third-party sources that corroborate what you claim.

Should I block GPTBot? That is a separate decision from citations. GPTBot crawls for model training; OAI-SearchBot and ChatGPT-User handle search and live retrieval. You can disallow the training crawler and still be cited in ChatGPT answers, provided the retrieval agents are allowed.

Does Cloudflare block AI crawlers by default? Cloudflare offers one-click AI crawler blocking and has moved toward blocking AI training crawlers by default for new zones. Because the setting can be active without anyone choosing it, verify with a live request rather than assuming your configuration is permissive.

Why does robots.txt allow the crawler but it still can't reach my site? robots.txt is advisory; your WAF, CDN or security plugin enforces separately and doesn't consult it. A crawler blocked at that layer receives a 403, a 429, or a JavaScript challenge. Test with a real request carrying the crawler's user-agent — that's the only check covering both gates.

Does allowing AI crawlers hurt my SEO or my bandwidth? It has no effect on Google rankings — Googlebot is a separate agent you already allow. Bandwidth impact is small for a normal site because these crawlers fetch far less aggressively than a full search index refresh. If volume is a genuine concern, rate-limit rather than block.

How do I tell whether an AI crawler is really who it claims to be? User-agent strings are trivially spoofed. The operators publish IP ranges for their crawlers, and several support HTTP message signatures for cryptographic verification. Allow-list on verified IP or signature so you can permit legitimate crawlers without permitting anything that copies the header.

How often should I re-check crawler access? At least quarterly, and after any CDN, host, plugin or WAF change. Access regressions are silent — nothing breaks visibly on your site — so continuous monitoring is materially better than periodic manual checks.