Article contents
Generative engines do not rank pages, they choose sources
A traditional search engine produces an ordered list: ten results, a rank for each, and a contest for the top three positions. A generative engine produces a text. To write it, it must first gather material, and that is where everything is decided: it keeps only a handful of sources, extracts passages from them, then writes an answer that cites them.
The mechanism, known as retrieval-augmented generation (RAG), always follows the same sequence. The user's question is rewritten into several sub-queries, often three to ten, which cover the implicit angles of the request. Each sub-query interrogates an index: Google's for AI Overviews and AI Mode, a proprietary or partner index for ChatGPT, Claude and Perplexity. The engine retrieves passages, not whole pages, then the model synthesises them and attributes the claims to their sources.
- 01QuestionThe user phrases a request in natural language, often long and contextualised.
- 02Sub-queriesThe engine breaks it down into 3 to 10 searches that cover the implicit angles.
- 03RetrievalEach sub-query brings back passages, not pages, from an index.
- 04Selection3 to 8 sources are kept: accessible, precise, identified, corroborated. Everything is decided here.
- 05Cited answerThe model writes and attributes each claim to its source.
Three practical consequences follow from this. First: being seventh on Google does not protect you, and being first guarantees nothing, because selection happens at passage level, not URL level. Second: a very precise page on a narrow question can be cited ahead of a more powerful but vaguer site. Third: the number of places is small. An answer generally cites between three and eight sources; you are no longer fighting for a position, but for a place on a short list.
Four conditions, in this order
In the generative visibility audits we carry out, cited pages share four properties, and absent pages almost always lack at least one of them. These conditions are sequential: if the first is not met, the other three do not count, whatever the quality of the content.
- 1AccessibleAI search crawlers can read the page's full HTML, with no blocking and no deferred rendering.Otherwise: the page does not exist for the engine.
- 2CitableEach section answers a question on its own, with evidence, in forty to eighty words.Otherwise: no passage is extracted.
- 3IdentifiedThe entity speaking is described, marked up and consistent wherever it appears.Otherwise: the source is judged unreliable.
- 4CorroboratedIndependent sources say the same thing about you, in the same words.Otherwise: your claim remains an opinion.
The rest of this article details each condition, with the checks to run and the fixes to make, then the measurement method that tells you whether the work is paying off.
Condition 1: being accessible to generative engine crawlers
Each model provider operates several crawlers, and they do not have the same role. Some collect training data, others feed the search index, and others still fetch a page at a user's request. Blocking the wrong crawler amounts to removing yourself from the answers.
| Provider | Agent | Role | Effect of blocking |
|---|---|---|---|
| OpenAI | GPTBot | Model training | None on ChatGPT search |
| OpenAI | OAI-SearchBot | ChatGPT search index | Disappearance from ChatGPT answers |
| OpenAI | ChatGPT-User | Fetching at a user's request | Page unreadable when a user asks for it |
| Anthropic | ClaudeBot | Model training | None on Claude search |
| Anthropic | Claude-SearchBot | Claude search index | Disappearance from Claude answers |
| Anthropic | Claude-User | Fetching at a user's request | Page unreadable when a user asks for it |
| Perplexity | PerplexityBot | Perplexity index | Disappearance from Perplexity answers |
| Perplexity | Perplexity-User | Fetching at a user's request | Page unreadable when a user asks for it |
Googlebot | Classic index, AI Overviews, AI Mode | Disappearance from Google, AI answers included | |
Google-Extended | Gemini training only | None on AI Overviews or AI Mode | |
| Microsoft | Bingbot | Bing index, Copilot and partners | Disappearance from Copilot and from the engines that rely on Bing |
Table: scroll horizontally.
The most common confusion concerns Google. The Google-Extended token only controls the use of your content for training Gemini; it has no effect on your presence in AI Overviews or AI Mode, which rely on Googlebot and on the classic index. Conversely, blocking OAI-SearchBot or PerplexityBot removes you from ChatGPT search and from Perplexity, even if your pages remain perfectly indexed by Google.
Blocking is not always deliberate. Many sites have enabled, on their CDN or their firewall, a “block AI bots” rule that treats all these agents the same way. Others return a 403 error to requests without browser headers, or hide the content behind a consent banner rendered server-side. In our audits, this accidental blocking is the leading cause of absence from generative answers on sites that are otherwise well ranked.
The checks to run
- Read the
robots.txtfile and explicitly distinguish training crawlers from search crawlers. The example below allows search and refuses training; adapt it to your own policy. - Review the rules of the CDN, the firewall and the hosting provider: lists of blocked agents, rate limiting, JavaScript challenges imposed on non-human requests.
- Search the server logs for the agents in the table. No visit in thirty days means an upstream block, or a lack of discovery.
- Check that the main content is in the HTML returned by the server. AI search crawlers execute little or no JavaScript: a page whose text only appears after client-side rendering is, for them, an empty page.
# Search crawlers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Training crawlers: separate decision (here, refused)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Condition 2: writing pages a model can cite
Once the page is accessible, the engine looks in it for passages able to answer a sub-query on their own. A citable passage has a recognisable shape: it fits in forty to eighty words, it starts with the answer, it contains at least one verifiable element (a figure, a date, a name, a condition) and it remains understandable out of context.
In practice, this shape changes the way a page is written.
- Headings take the form of the questions people actually ask. “How long does it take to get cited by ChatGPT?” will be aligned with a sub-query; “Our timelines” never will.
- The first paragraph under each heading answers directly. Context, history and nuance come afterwards. A model that has to read three paragraphs before finding the answer moves on to the next source.
- Every important claim carries evidence. A dated figure, a named source, a described method. Vague wording (“fast results”, “recognised expertise”) is never extracted, because it answers nothing.
- Tables, lists and definitions are favoured. These formats are reused almost as they are, and they reduce the risk of the model rephrasing your point inaccurately.
- The update date is visible and honest. On topics that evolve, generative engines favour recent sources; a page with no date, or dated 2022, starts at a disadvantage.
The second wording is citable: it answers a precise question, it contains a timeframe and two conditions, and it remains true out of context. The first says nothing an engine could reuse.
Condition 3: making your entity legible
A generative engine does not only cite a text, it cites someone. Before keeping a source, it tries to establish who is speaking, in what capacity, and whether that identity is consistent with what it finds elsewhere. This reading goes first through structured data, then through the consistency of the information about your company wherever it appears.
- Organization: legal name and trading name, address, identifiers, and above all the
sameAsproperty, which links your site to your official profiles (LinkedIn, company register, Wikidata, sector directories). - Person for each author: role, affiliation, publications, profiles. An author page that documents verifiable expertise carries more weight than an anonymous byline.
- Article or BlogPosting: author, publication and modification dates, publisher, main image.
- FAQPage for question-and-answer sections: the format most directly aligned with the question-and-answer workings of generative engines.
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "SEO360",
"legalName": "SEO 360 ALICANTE S.L.",
"url": "https://seo360.tech",
"sameAs": [
"https://www.linkedin.com/company/…",
"https://www.wikidata.org/wiki/…"
],
"founder": {
"@type": "Person",
"name": "Kamel Malek"
}
}
Consistency matters as much as markup. If your site says “SEO360”, your LinkedIn page “SEO 360 Alicante” and a directory “Seo360 Agency”, the model reconstructs three weak entities instead of one strong one. Harmonising names, descriptions and contact details across all your external presences is often the least costly lever in the whole programme.
The llms.txt file, proposed as a site summary intended for models, belongs to a different logic. It takes an hour to produce and does no harm, but no engine provider has documented its use in source selection. Treat it as a complement, not as a lever.
Condition 4: being corroborated somewhere other than your own site
A model that has to recommend a provider, a product or a method looks for consensus. A claim found only on your own site is an opinion; the same claim repeated by a media outlet, a sector directory, a comparison site, a forum or a publication becomes information. This is why answers to “best”, “comparison” or “which one to choose” queries overwhelmingly cite third-party pages rather than the sites of the brands concerned.
Corroboration is built as a presence, not as a link campaign. The sources that generative engines draw on most often are:
- press articles and trade publications that describe your company in the same terms as your site;
- the comparison sites and specialist directories of your sector, including those that place you among others;
- forum and community discussions, where a useful answer signed by an expert from your team is worth more than a promotional mention;
- detailed customer reviews, on platforms the engine can read;
- audio and video content whose transcript is accessible, as well as contributions from identified authors.
The key word is “mention”, not “link”. A link remains useful for Google; for a generative engine, it is the consistent repetition of your name, your speciality and your evidence across independent sources that makes the difference.
Measuring your citation share
Generative visibility cannot be read in a rankings report. The indicator that replaces it is citation share: on a panel of questions your buyers actually ask, the proportion of answers in which your brand or your domain appears, engine by engine. The method has four steps.
- Build a panel of thirty to a hundred questions, split between informational questions (“how…”), comparative questions (“X or Y”) and transactional questions (“which provider for…”), in every language you sell in.
- Run the panel every week on ChatGPT, Perplexity, Gemini and Claude, recording the sources cited and the brands named. Answers vary from one run to the next; each question is run several times and an average is tracked.
- Calculate the citation share for you and for your three direct competitors, by engine and by intent type.
- Link it to revenue: referral traffic from the engines' domains in your analytics tool, changes in searches for your brand, assisted conversions over the period.
Illustrative data. The useful reading: this brand is competitive on Perplexity, absent from Google AI Overviews, and its competitor dominates ChatGPT; the action plan follows engine by engine.
Two precautions. A one-off measurement means nothing, because answers fluctuate; it is the trend over several weeks that counts. And citation share must be read by intent: being cited on definition questions is worth nothing if you are absent from provider-choice questions.
The mistakes that cost citations
The causes of absence we encounter most often come down to seven points. Each can be checked in under an hour.
- A blanket anti-bot rule that blocks
OAI-SearchBot,PerplexityBotorClaude-SearchBotalong with the training crawlers. - Main content injected by JavaScript, invisible in the initial HTML.
- Pages with no date, or whose update date is artificially refreshed without any change to the content.
- Generic presentation copy, with no figure, no name and no method.
- An entity that is inconsistent from one platform to another: names, descriptions and contact details that diverge.
- No presence on the third-party sources the engine consults for your sector.
- Reporting limited to Google positions, which shows neither the problem nor the progress.
A ninety-day action plan
The programme below is the one we run on a typical engagement. It starts with access and measurement, because there is no point rewriting pages the engines cannot read, nor acting without a quantified starting point.
- Weeks 1 to 2Access and measurement
- Audit of robots.txt, CDN, firewall, HTML rendering
- Panel of 30 to 100 questions per language
- First citation share measurement, by engine
- Weeks 3 to 6Pages and entity
- Rewriting the ten priority pages as direct answers
- Organization, Person, Article, FAQPage structured data
- Author pages and harmonisation of external presences
- Weeks 7 to 12Corroboration and iteration
- Third-party mentions: press, directories, comparison sites, communities
- Second measurement and reading by intent
- Prioritising the next pages according to the gaps observed
At the end of the twelve weeks, the goal is not to have “finished”, but to have installed a loop: weekly measurement, rewriting of absent pages, continuous corroboration. Generative engines change fast; an organisation that measures every week sees the drops before they cost anything, and the gains before its competitors do.
What to remember
- A generative engine selects three to eight sources per answer, at passage level, not page level.
- Four conditions, in order: accessible to AI search crawlers, citable, identified, corroborated.
- Distinguish training crawlers from search crawlers: blocking the latter erases you from the answers.
- A citable passage fits in forty to eighty words, starts with the answer and contains evidence.
- The steering indicator is citation share, measured every week, by engine and by intent.
Frequently asked questions
Should you block GPTBot to protect your content?
That is an editorial choice with no effect on your visibility in ChatGPT search. GPTBot is used for model training; search relies on OAI-SearchBot and on ChatGPT-User. You can refuse the first and allow the other two. The only mistake is to block everything with a single rule.
How long does it take to appear in AI answers?
On our engagements, the first citations generally appear six to ten weeks after structured data is in place and the main pages have been rewritten, provided AI search crawlers have access to the site. Topics dominated by third-party sources take longer, because those sources must first be reached.
Is traditional SEO enough to get cited by generative engines?
No, but it is the foundation. Technical quality, authority and semantic clarity remain necessary. Generative engines additionally require direct, self-contained answers, an entity clearly identified through structured data, and corroboration by independent sources. A page that ranks well on Google but is written as promotional copy is almost never cited.
Is the llms.txt file necessary?
No. No engine provider has documented its use in source selection. It does no harm, costs little, and can serve as a structured summary of your site; but it replaces neither crawler access, nor citable pages, nor correct structured data.
