Short answer, as of August 2026: each major AI engine runs its own crawler under its own user agent, and you grant or deny each one separately. That single fact explains most "why am I cited here but not there" confusion. Before theorising about content quality, check whether GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, Applebot-Extended and Bingbot can each actually reach your pages — because a site can be perfectly optimised and still be invisible on three of six surfaces.
Here's what governs each one, and what to do about it.
The shared mechanism
Strip away the branding and every citation-producing AI answer works roughly the same way: the system decides it needs current information, issues one or more retrieval queries against an index, fetches a handful of candidate documents, and composes an answer over them with attribution.
Three consequences follow, and they apply universally:
- If a document is not in the index, it cannot be cited. Retrieval precedes generation. This is why "just do good SEO" is unhelpful advice that happens to be correct — findability is a hard prerequisite, not a nice-to-have.
- The model must be able to lift something. Retrieved documents get read for extractable propositions. Prose with no specifics gives the model nothing to attribute, so it summarises a competitor instead.
- Corroboration beats assertion. When candidate documents disagree, models hedge. Consistent facts across independent sources are what license a confident, named answer.
Those three principles are the whole discipline. The engine-by-engine differences below are about access and coverage, not about different optimisation strategies.
What each engine's crawler is called
This is the practically important table, because these strings are what you put in robots.txt.
| Engine / surface | Crawler user agents | What blocking it costs you |
|---|---|---|
| ChatGPT | GPTBot (training), OAI-SearchBot (search), ChatGPT-User (user-initiated fetch) | Separate purposes — blocking all three removes you from ChatGPT's cited sources |
| Perplexity | PerplexityBot | Perplexity cites heavily and visibly; a block is highly costly here |
| Claude | ClaudeBot, anthropic-ai | Absence from Claude's web-search citations |
| Google (Gemini, AI surfaces) | Google-Extended — distinct from Googlebot | You stay in classic Search but opt out of Gemini-side use |
| Microsoft Copilot | Bingbot primarily | Blocking Bingbot damages Copilot and Bing simultaneously |
| Apple Intelligence | Applebot, Applebot-Extended | Apple-surface visibility |
| Downstream / open datasets | CCBot (Common Crawl) | Feeds many third-party systems indirectly |
Two details cause most of the accidental damage:
GPTBot and OAI-SearchBot are different agents with different jobs. OpenAI documents them separately (platform.openai.com/docs/bots). Blocking GPTBot because you don't want your content used for training, while leaving OAI-SearchBot allowed, is a coherent and increasingly common position. Blocking both because someone pasted a "block AI" snippet from a forum in 2023 is not a decision — it's an accident, and it's the single most common cause of total ChatGPT invisibility.
Google-Extended is not Googlebot. Disallowing Google-Extended does not remove you from Google Search; it opts you out of Gemini-side usage. Plenty of sites set this during a cautious legal review and never revisited it, then wondered why they show up in classic results and never in AI ones.
Go read your own robots.txt right now. In roughly half the audits we see, at least one of these is closed by accident.
Why coverage differs even when access is open
Access is necessary, not sufficient. Engines diverge on three further axes.
Index provenance. Engines draw on different underlying indexes — some their own crawl, some a partner search index, some a blend. This is why a page can be freshly indexed by one system and unknown to another for weeks. It's also why Bing-side health matters more than most SEO teams assume: several assistant surfaces lean on Bing's index, so a site neglected in Bing Webmaster Tools is quietly handicapped on more than one AI surface.
Recency behaviour. Some surfaces strongly prefer recently-crawled documents; others happily cite year-old pages if they're the best available. Practically: a lastmod that reflects real substantive updates, and a sitemap that matches your canonicals byte-for-byte, help more here than they do in classic SEO. Google's sitemap documentation covers the format; the failure mode that hurts here is a canonical pointing at a URL that redirects.
Citation style. Perplexity attributes inline and densely — it is the most legible engine to optimise for, because you can see exactly which passage it lifted. Conversational assistants may name a business without linking at all. That difference matters for measurement: on some surfaces you count citations, on others you count mentions, and conflating the two produces reports nobody can act on.
What this means practically
Audit access per engine, not once. "We allow AI crawlers" is not a finding. The finding is a seven-row table with a yes or no in each cell, verified against the live file rather than the version in your repo — those diverge more often than anyone expects, especially where a CDN injects or rewrites robots.txt.
Check the layer below robots.txt. A WAF, bot-management rule or aggressive rate limit can block a crawler that robots.txt explicitly welcomes. Fetch your own key pages with each user agent string and confirm you get real HTML, not a challenge page. This is the check teams skip, and it's the one that explains "we allowed everything and nothing changed."
Serve the substance in the initial HTML. Fetch-only crawlers don't run your JavaScript. Bing's webmaster guidelines have said this for years and it has never been more consequential — a client-side-rendered page is functionally empty to several of the agents in the table above.
Measure per engine, separately. Aggregate "AI visibility" hides the actionable signal. Cited by Perplexity and Claude but never Gemini is a diagnosis: probably Google-Extended, possibly index freshness. Cited by nobody is a different diagnosis entirely. Our diagnostic for when competitors are cited and you aren't walks the full decision tree.
The content side, which is engine-agnostic
Once access is open everywhere, the content standard is the same across all six surfaces — which is a relief, because six separate content strategies would be unmanageable.
The Princeton-led generative engine optimization study tested content modifications against a benchmark of generative-engine responses and found that adding citations, statistics and direct quotations raised source visibility by up to 40%, while keyword stuffing did nothing. That result has held up as a working standard: write things a model can extract as standalone true statements.
In practice that means specific numbers over adjectives, named exceptions over blanket claims, real external authorities cited for factual assertions, and the literal question answered in the first two sentences of a section rather than in paragraph four. Google's guidance on creating helpful, reliable, people-first content points at the same target from the classic-search direction, which is convenient: one standard serves both. Our fuller treatment of how to get cited by ChatGPT and Perplexity works through the editorial side in detail.
If you want a concrete way to check a draft against it before publishing, scoring a draft for AI visibility covers the review pass.
Structured data's role across engines
Structured data doesn't make an engine cite you. It does something more specific: it makes your business a resolvable entity — an unambiguous thing with a name, a category, a location and a set of properties that a system can match against other sources.
That matters because corroboration is what turns a hedge into a named recommendation. Use Schema.org vocabulary, keep it consistent with the visible page, and — critically — generate it from the same canonical record the page body renders from. Hand-authored markup drifts the first time a fact changes, and markup that contradicts the page is worse than no markup at all.
For businesses with multiple locations this compounds fast, and the architecture that survives it is in the multi-location SEO playbook. For a sector-by-sector view of how much any of this moves the needle, how GEO differs for local services, SaaS and ecommerce is the more relevant read.
Tracking six surfaces without losing your mind
The measurement problem is arithmetic. Twenty prompts, six engines, three repetitions for non-determinism is 360 checks per cycle, per brand. Manually that's a task that gets skipped the first busy week; the engine-level breakdown — the actually useful part — is the first thing to go.
DigiRank's AI Visibility Tracker samples prompt sets across ChatGPT, Gemini, Perplexity, Claude, Grok and Microsoft Copilot on a recurring schedule and reports per-engine share of voice over time, so "we're invisible in Gemini specifically" shows up as a line on a chart instead of a hunch. Plans start at $99/mo, with the $249/mo Agency tier covering multiple client tenants and a 14-day trial, and it connects to Search Console, GA4 and Google Business Profile so per-engine AI data sits alongside conventional performance.
Start with the robots.txt table, though. It's free, it takes ten minutes, and it's the most likely explanation for whatever asymmetry sent you looking.
Frequently asked questions
Why does Perplexity cite my site but ChatGPT never does?
Almost always a crawler access difference. PerplexityBot and OpenAI's GPTBot / OAI-SearchBot are separate agents governed separately in robots.txt, and a bot-management rule at your CDN can block one while allowing another. Index coverage and recency differences account for the rest.
What's the difference between GPTBot and OAI-SearchBot?
They serve different purposes and OpenAI documents them as distinct agents. Blocking GPTBot while allowing OAI-SearchBot is a deliberate, coherent position — declining training use while remaining eligible as a cited search source. Blocking both, usually by pasting a generic "block AI" snippet, removes you from ChatGPT's cited sources entirely.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended is separate from Googlebot. Disallowing it opts you out of Gemini-side usage while leaving classic Search unaffected — which is exactly why sites that set it during a legal review later find themselves in normal results but never in AI ones.
Do I need a different content strategy for each AI engine? No. Access controls differ per engine; the content standard does not. Specific sourced claims, statistics, quotations, named exceptions and direct answers early in each section serve every surface, and they align with what classic search rewards too.
Why does Bing matter for AI visibility? Several assistant surfaces, Microsoft Copilot most directly, lean on Bing's index. A site that is healthy in Google Search Console but neglected in Bing Webmaster Tools is handicapped on more than one AI surface at once, usually without anyone noticing.
Should I track AI visibility per engine or as one number? Per engine. An aggregate score hides the diagnosis — being cited by Perplexity and Claude but never Gemini points at a specific, fixable cause, while a single blended percentage tells you nothing you can act on.
Does structured data make an AI engine cite me? Not directly. It makes your business a resolvable entity that systems can match and corroborate across sources, which is what converts a hedged answer into a named recommendation. Generate it from the same record the page renders from so it can never contradict the visible content.