HOW GOOGLE’S SEARCH PIPELINE WORKS: FROM RETRIEVAL TO RANKING & AI ANSWERS

Most companies investing in organic visibility are optimising for Google search ranking and skipping the stage that determines whether their content enters the competition at all. I’m Szymon Słowik, SEO consultant and strategist, founder of takaoto.pro and creator of the BUXS Framework. I’ve spent 14+ years consulting SEO and organic growth for companies like SMSAPI Brand24, Santander, and eSky Group, and I’ve presented variations of this thinking at BrightonSEO, Chiang Mai SEO Conference and Baltic SEO Summit.

This article maps how information retrieval and ranking systems in Google process a user’s query through query rewrite, retrieval, ranking, and AI answer assembly. It frames what each stage means for how you invest budget, structure content architecture, and measure returns on the search experience and organic visibility.

Google and other modern search engines run a multi-stage pipeline: information retrieval (candidates selection using inverted indexes and vector embeddings), web ranking (machine-learned scoring, NLP-based re-ranking, behavioural signal adjustments), and, increasingly, AI interpretation (confidence-weighted passage arbitration and answer synthesis through Retrieval-Augmented Generation). If your content is not eligible for retrieval, ranking work is irrelevant. It doesn’t compete.

Understanding this pipeline is what underpins Semantic Retrieval Optimization (SRO), an approach I know from Sergey Lucktinov and Koray Tuğberk Gübür. I apply SRO in every strategic engagement to structure content so that it is eligible for retrieval, competitive in ranking, and citable by AI.

Szymon Słowik is an SEO consultant, strategist, and creator of the BUXS Framework (Brand × UX × Semantics), a methodology for structuring organic growth around retrieval eligibility, semantic architecture, and brand-level trust signals. He advises B2B, SaaS, e-commerce, and enterprise organisations on search strategy, AI search visibility, and content architecture. Founder of takaoto.pro. Speaker at BrightonSEO, Baltic SEO Summit, Chiang Mai SEO Conference, and SEO Estonia.

TL;DR for business, marketing and strategy leaders

This is a pillar page defining how Google’s search pipeline selects information. It explains the query rewrite → information retrieval → ranking → AI arbitration sequence and why visibility now depends on eligibility before position on the search engine results page. If you invest in SEO, AI search visibility, or content strategy, this is the structural layer your decisions sit on.

• Google’s ranking systems only score content that has already been retrieved. Most SEO ignores the retrieval gate entirely.
• AI search systems evaluate passages, not pages. The unit of competition shifts from position to citation.
• Visibility now depends on entity-level clarity, structural extractability, and confidence level across both web ranking and AI answer assembly.

What you will learn

1. How information retrieval filters content before Google’s ranking systems begin scoring, and why most SEO budgets miss this gate.
2. How BM25 lexical matching and neural embeddings generate retrieval candidates in parallel, and what that means for your content strategy.
3. What Navboost, BERT, twiddlers, and site quality score do inside Google’s ranking pipeline.
4. How AI search systems (AI Overviews, Perplexity, ChatGPT) use query fan-out and confidence-weighted arbitration to select passages for grounded answers.
5. How to diagnose whether your binding constraint is retrieval, ranking, or AI eligibility, and where to put the next euro.
6. Why semantic architecture compounds while tactical SEO delivers linear returns.

If your content is not architected for information retrieval and AI selection, no amount of ranking work will make it visible.

Most SEO advice starts at ranking. That’s the wrong entry point.

Ranking models evaluate only the content that made it through retrieval. If a page fails the retrieval gate because of render issues, semantic ambiguity, entity confusion, or structural problems, it never enters the candidate set. It changes where you put budget, how you structure information architecture, and what you report to the board.

Ahrefs’ research found that 96.55% of web pages get zero traffic from Google. Most of that failure likely happens at retrieval, not ranking. The content exists in the index. Absent from the competition.

If you are a founder or CMO at a scaling B2B SaaS or tech company spending €50K to €200K per year on SEO, the Google pipeline is the lens through which every investment decision should be evaluated. Not “are we ranking?” but “are we even being retrieved, and for which queries?”

What happens when links grow but traffic doesn’t?

In a recent audit for a boutique M&A advisory firm, the pattern was textbook. They had invested in keyword targeting and link acquisition. New referring domains were climbing steadily, but organic visibility stayed flat and even decreased lately. No ranking movement. No traffic growth.

The links were not broken. Content was not thin. The problem was upstream.

Their “About Us” page was interchangeable with any consulting firm in any vertical: no named founders, no credentials, no case studies, no entity signals connecting the brand to the people behind it. Service pages described what M&A advisory is, not what this firm does differently or for whom. From the retrieval system’s perspective, there was nothing to disambiguate. No reason to prefer this domain over any other candidate making the same generic claims. Ever seen a site like that? I see it constantly.

We are now repurposing their content with four main interventions: proper ICP framing on every service page (who this firm serves, what constraints those clients face, what outcomes look like), structured data connecting the brand entity to its founders and their credentials, topical coverage filling the gaps between existing pages, and content enrichment for better contentEffort scoring and information gain.

The goal isn’t just more content. It’s making what they already have retrievable by increasing E-E-A-T signal density, resolving entity ambiguity, and closing the semantic loops that retrieval systems use to build confidence. Links amplify a signal the system trusts. Without that trust foundation, they were amplifying nothing.

What are the stages of Google’s search pipeline?

Google’s pipeline processes content and queries through two parallel tracks that converge at retrieval. The content processing track (crawl → data processing → indexing and embeddings → data stores) prepares documents for the system. The query processing track (query → query rewrite → retrieval → ranking → output) handles the user’s query in real time. Here is the simplified sequence:

1. Crawling and ingestion: how Google discovers new pages by following links from known pages and sitemaps, downloading them via Googlebot and other automated programs called crawlers, or ingesting structured data from other sources.
2. Indexing and embeddings: parsing, render execution, and a two-stage process. Content is first parsed and rendered, then classified thematically into topical corpora using methods like TF-IDF and semantic analysis, and stored in the Google index: a massive database split between the inverted index (for lexical retrieval) and vector embeddings (for neural retrieval).
3. Query rewrite: interpreting what happens when a user enters a query, through intent classification, query expansion, query augmentation, and entity type classification before retrieval begins. Navigational queries may bypass the pipeline entirely and route to personalised browse.
4. Information retrieval: candidates selection using BM25 lexical matching and neural matching (vector embeddings) to select retrieved documents that could satisfy a specific query after rewriting.
5. Ranking: machine-learned scoring, NLP re-ranking via BERT and transformer models, signal-based adjustments (Navboost, twiddlers), and personalised event-based ranking to order candidates by predicted relevance.
6. Output: two distinct surfaces. Traditional web ranking produces the final ranking as a personalised search engine results page containing organic results, ads, featured snippets, and other features, all assembled and scored by systems including Navboost and Glue. AI answer assembly produces a grounded AI answer with source citations via Retrieval-Augmented Generation (RAG).

Each stage is a filter. Content that fails any stage is invisible to the next. The strategic question is: which stage is your binding constraint?

What is information retrieval in Google’s pipeline and why does it precede ranking?

Information retrieval is the candidates selection stage, the retrieval process where Google narrows its index of roughly 400 billion pages (a figure referenced in Google’s antitrust trial testimony) down to 1,000 to 10,000 potentially relevant documents that could satisfy a user’s query. Scoring every page against every query is computationally impossible, so Google filters first, discarding irrelevant results before ranking begins. Retrieval is a gate, not a scoring curve.

If a page doesn’t pass retrieval, it doesn’t compete. Backlinks, authority, production cost: none of it matters if the page never enters the candidate set. Think of it as a procurement shortlist, not a performance review.

Information retrieval is a well-defined pattern in computer science: a user interface sends a query to a search engine, which scores candidates against a distributed database, applies ranking logic, and returns results. The patent diagram below (US9405794B2, ThoughtSpot Inc.) shows this pattern in an enterprise business intelligence context, but the components map directly to how Google’s pipeline works: a search engine sitting between user queries and a distributed in-memory database, with cluster management handling scale.

For a business investing in content production, this has a direct cost implication. Every page that fails retrieval eligibility is sunk cost. Writer time, editorial cycles, design resources, all producing assets that never enter the competition. I’ve run audits where 60% or more of a site’s published pages had zero impressions in Google Search Console for any target query. Not low rankings. Zero impressions. The pages existed in the index but were absent from candidates selection entirely. Ever checked your own site for that pattern?

What determines whether Google retrieves a page?

Three interdependent layers determine whether a page enters the retrieved documents set. Each must be satisfied. The evaluation is entirely query dependent: failure at any single layer eliminates the page from the candidate set for that query.

Layer 1: Technical accessibility. If Googlebot cannot render the page, or if performance falls outside acceptable thresholds, the content doesn’t exist from the system’s perspective. TTFB under 500ms, LCP under 2.5s on mobile, clean DOM under 1,500 nodes, no accordion-hidden primary content, no render-blocking JavaScript that prevents content parsing. Technical compliance is necessary, not sufficient. But without it, the other layers become irrelevant.

Layer 2: Entity-level clarity. Does the content explicitly address the entities and relationships the user’s query implies? A SaaS company’s “platform overview” page that never mentions specific capabilities, integration types, or use-case contexts by name might be indexed but never retrieved for the queries its potential customers search. Indexed, yes. Retrieved, no.

Entity type classification is part of this layer. Can Google confidently associate the page with the correct entities in its Knowledge Graph? If your brand, your product category, or your topic area creates disambiguation problems (multiple meanings, competing entities, unclear relationships), the retrieval system hedges. Hedging means exclusion. I deal with this on my own site: “Szymon Słowik” competes with at least four other Knowledge Graph entities, including a Polish soldier with a Wikipedia entry. Without explicit entity signals on every page, Google wouldn’t know which Szymon Słowik to associate with SEO consulting content.

Layer 3: Trust and authority signals. Here the system evaluates whether a retrieved candidate deserves to compete. Navboost (behavioural data indicating user satisfaction), external corroboration (other authoritative sources referencing your content or brand), and link portfolio quality all contribute to what the DOJ trial documents revealed as a site-level trust assessment.

A domain with strong trust signals gets more generous retrieval; its pages could be included in candidate sets for a wider range of queries. A domain with weak or conflicting signals would face a narrower retrieval window. How many queries a domain’s pages are eligible for depends directly on the confidence level Google assigns.

These trust and quality signals are pre-computed and compressed into what the leaked Content Warehouse API reveals as CompressedQualitySignals, a per-document “rap sheet” that feeds the Q* scoring system. Q* calculates a composite quality score from link data, site structure, content quality metrics, and aggregated user behaviour. That score is loaded into a high-speed flash-memory cache called Mustang, which performs the actual first-pass gate during retrieval. Documents failing Mustang’s quality threshold are eliminated before any neural ranking model, any Navboost signal, or any semantic relevance score ever evaluates them. This is the single most important SEO implication of the pipeline architecture: if your Q* score, Panda demotion, or siteAuthority fails the threshold, no downstream system can save you.

Structural coherence operates across all three layers. Heading hierarchy, passage-level clarity, and internal linking all contribute to what I would frame as semantic cost: the computational effort the system expends to understand what a page is about. Lower semantic cost means higher retrieval probability. An engineering problem, not a copywriting problem.

This is where Brand, UX, and Semantics converge: the BUXS model I use as a diagnostic lens in every strategic engagement. Brand as entity affects trust signals and disambiguation. UX affects behavioural signals that Navboost feeds back into retrieval generosity. Semantic architecture affects entity-level clarity and reduces the cost of retrieval. Each dimension works independently, but they multiply each other’s impact. Hm, that might be the single most important sentence in this article.

How does Google generate retrieval candidates?

Google runs two retrieval systems in parallel, lexical matching through BM25 and neural matching through vector embeddings, and merges their candidate sets before passing retrieved documents to the ranking stage. Most SEO discussions treat this process as a black box. Content goes in, candidates come out. But the mechanism matters.

The graph from Google’s own Vertex AI Search documentation confirms this architecture: both retrieval methods draw from the same data stores, which are populated by the indexing and embeddings stage upstream.

How does BM25 select relevant documents from Google’s inverted index?

BM25 selects relevant documents by scoring term overlap between the user’s query and the inverted index, a reverse lookup table that maps every term in the corpus to the documents containing it. When a query arrives, Google identifies documents sharing terms with that query, then scores them using BM25: a probabilistic relevance function that weights term frequency, document length, and inverse document frequency.

BM25 is a sparse retrieval method. It matches on exact words and their known variants. At over 13 billion queries per day (DemandSage, 2025–2026 estimates), speed is not a preference. It’s a hard constraint. BM25 remains the backbone of candidates selection in Google and most search systems precisely because it scales under that load. Newer, fancier methods are available, but none of them can replace BM25 at this volume. Not yet.

Terminology precision matters at the information retrieval stage. If your B2B SaaS company calls its product a “workflow orchestration platform” but your prospects search for “project management software,” the lexical gap could prevent retrieval before any ranking model gets involved. I’ve seen this exact pattern across multiple SaaS audits. The team builds content around internal product language, and then wonders why organic traffic stays flat. Hm, and that’s an easy trap to fall into. Query-meaning-intent matching starts with using the words your audience types, not the words your product team prefers.

How do vector embeddings retrieve relevant documents by meaning?

Vector embeddings retrieve relevant documents by meaning rather than by terms. Documents and queries are converted into dense vector representations (embeddings) using transformer models. Retrieval becomes a geometry problem: find the document vectors closest to the query vector in embedding space.

Because searching billions of vectors exhaustively is too slow, systems use Approximate Nearest Neighbor (ANN) algorithms to find high-similarity candidates efficiently. ANN indexing trades a small amount of recall precision for massive speed gains, making neural retrieval viable at web scale. Google’s indexing methods have evolved to include large language models and vector databases for more sophisticated organisation of information. Research like MUVERA (Multi-Vector Retrieval via Fixed Dimensional Encodings, NeurIPS 2024) points to where this is heading: making multi-vector retrieval as fast as single-vector search while preserving richer semantic understanding. BlockRank (Gupta et al., 2025, Google-affiliated research) takes this further by applying structured sparse attention within LLMs to make in-context ranking scalable, meaning large language models could directly rank retrieved documents by exploiting learned attention patterns that correlate with relevance. The graph confirms that embeddings are created during the “Indexing and embeddings” stage and stored in the same data stores that feed lexical retrieval.

Neural retrieval captures semantic relevance that BM25 misses through semantic matching. A page about “reducing customer acquisition cost” would be retrieved for queries about “improving CAC efficiency” even if the exact terms don’t match, because the embedding vectors are close in the semantic space. That’s the mechanism behind what some in data science call “dense retrieval.” For practitioners, the takeaway is simpler: covering the conceptual neighbourhood of a topic increases the number of queries your content could be retrieved for, even queries you never explicitly targeted.

Why does Google run lexical and neural retrieval in parallel?

Google runs both systems in parallel because each catches what the other misses. Lexical retrieval is fast and precise but blind to meaning. Neural retrieval understands meaning but may miss exact-match terms.

Your content needs to satisfy both query based selection logics:

Lexical coverage: include the precise terms and entity names your audience uses. If the inverted index can’t find your page for a query, BM25 won’t score it.

Semantic depth: cover the conceptual neighbourhood so that neural retrieval associates your content with related queries. If your embeddings are distant from the query vector, ANN search won’t surface your page.

Information relevance depends on both. Content that excels at one but fails the other leaves retrieval probability on the table. Semantic SEO addresses both. It structures content around entities and relationships (serving neural retrieval) while maintaining terminology precision (serving BM25). It’s both, or you’re leaving retrieved documents to your competitors that should have been yours.

How do Google’s ranking systems order the retrieved documents?

Google’s ranking systems take the candidate set from information retrieval (typically 1,000 to 10,000 retrieved documents from combined lexical and neural retrieval) and score them in layers to determine position. A system called Superroot orchestrates this: it coordinates retrieval across multiple data stores (main inverted indexes, Knowledge Graph, specialist indexes, document embeddings) and then executes the multi-tiered ranking cascade. Google employs hundreds of specialised systems that evaluate over 200 ranking signals to produce those positions. Most SEO discussions live at this stage. So do most budgets.

But Google search ranking isn’t a single algorithm. It’s a layered system: initial general ranking via lightweight models that assign relevance scores, then progressively deeper re-ranking via NLP transformer models (including BERT and its successors) that evaluate query-document relevance at the token level. The Vertex AI Search architecture confirms this layered design: signal enrichments from data stores feed into the ranking stage alongside personalised, event-based ranking from user events.

What is Navboost and how do user signals affect Google search ranking?

Navboost is Google’s behavioural ranking system that uses aggregated user interactions (click patterns, dwell time, navigation behaviour) to adjust web ranking based on real user satisfaction signals. It isn’t a simple click-through-rate metric. It’s a behavioural model trained on approximately 13 months of click logs, as revealed in DOJ antitrust trial testimony (document PXRD003, Durrett testimony demonstrative), that adjusts the ranking of URLs based on whether users found what they were looking for. Pandu Nayak’s interview notes (PXR0357) further confirmed the weight of interaction signals in the ranking pipeline.

Google’s own documentation labels this input as “personalized, event-based ranking” flowing from “User events” into the Ranking stage. Real behaviour from search interactions and potentially Chrome browser usage data, aggregated over months, feeding directly into how Google orders results.

Scoring documents based on historical data isn’t new. Google patent US7346839B2 (“Information retrieval based on historical data,” filed 2003, inventors include Matt Cutts and Jeff Dean) describes the three-step pattern: identify documents, obtain history data associated with those documents, then score them based at least in part on that history data. NavBoost applies the same logic to click history. The patent formalises it for link growth patterns, content change frequency, and user interaction trajectories over time. A document’s history isn’t passive context. It is an active scoring input.

Let me give you a concrete scenario. A CMO searches “SEO for SaaS” and lands on a page that talks about SEO generically without addressing SaaS-specific constraints like CAC pressure, long sales cycles, and multi-stakeholder buying committees. That is a dissatisfaction signal Navboost records. Page matched the query. It didn’t match the user intent or search intent. Navboost doesn’t care about your keyword targeting. It cares about whether the user stopped searching. And these engagement metrics compound over time, so one bad session doesn’t kill you, but a pattern of quick returns to SERP will.

UX is not a nice-to-have in web ranking. It is a weighted variable in the scoring function.

More context: US and Plaintiff States v. Google LLC, DOJ case page

How do BERT and NLP transformer models re-rank results for relevance?

BERT and subsequent NLP transformer models re-rank search results by evaluating the semantic relationship between the user’s query and each retrieved document at the token level, far deeper than BM25’s term matching. Google’s integration of BERT (Bidirectional Encoder Representations from Transformers), now applied to nearly every English-language query, uses deep learning to assess relevance through contextual understanding rather than keyword overlap, surfacing more relevant results.

Where BM25 matches terms, BERT understands context. The phrase “apple support” near “phone” resolves to Apple Inc. tech support, not fruit assistance. Content that is semantically precise, where entity relationships are explicit and unambiguous, would score higher than content relying on keyword density without contextual clarity.

Google’s ranking systems reward semantic precision, not keyword repetition. If your page covers the right topic but the entity relationships are vague or the language is ambiguous, BERT could score a competitor’s clearer page higher even if yours has stronger link signals. I’ve seen this happen with pages that rank on page two for months despite solid referring domains. The links are there. Semantic precision isn’t.

What are twiddlers and how does personalisation adjust Google ranking?

Twiddlers are modular re-ranking functions in Google’s pipeline that adjust scores after the initial ranking pass. They can boost or demote results based on freshness, content type, location, language, and dozens of other contextual variables.

Personalisation operates after retrieval, not before. A critical distinction, and the graph confirms it: “User events” feed into ranking, not into retrieval. The candidate set is assembled without personalisation. Then user-specific signals (search history, the user’s location, language preferences) adjust the ranking of already-retrieved candidates.

Personalisation doesn’t determine whether your content competes. It influences where it is positioned within the competition. Retrieval eligibility remains the universal gate. You might see different positions for the same query depending on context, but you won’t be invisible to one user and visible to another. Either you are in the candidate set, or you are not.

SERP positions aren’t deterministic. The question is not “what is the ranking factor?” It’s “which combination of signals, in which context, for which query class, could produce highly relevant results and the outcome we are investing towards?”

What is Google’s site quality score and why does brand as entity matter?

Google’s site quality score is a domain-level assessment that influences how all pages on a domain are ranked. The Google Leak materials confirmed attributes feeding into this score. Content quality signals (historically addressed by the Panda system), link spam detection via the Penguin system and SpamBrain, external link quality, user engagement, and E-E-A-T all contribute at the site level. Since 2022, the Helpful Content System specifically prioritises original content written for people rather than content created primarily for search engine traffic.

Brand as entity becomes a strategic asset at this level, and I don’t mean “brand awareness” in the vague marketing sense. I mean brand as a measurable input to a scoring system. A strong site quality score acts as a rising tide; it improves the baseline ranking position of every page on the domain. Conversely, domain-level quality problems could suppress even excellent individual pages.

Sometimes the highest-ROI SEO investment is pruning, not producing. Removing content that degrades domain signals and keeping only the important pages might unlock performance for the content that deserves to compete.

I did exactly this when I migrated my Polish site from a *.pl ccTLD to a /pl/ subcatalogue on szymonslowik.com. I cut a pretty big chunk of pages that were not relevant anymore, redirected what needed redirecting, and let the rest go. The domain got cleaner. Signals got stronger. The principle holds: a smaller, more coherent domain may outperform a larger, diluted one in Google’s ranking systems.

How do AI search systems and LLMs select passages for grounded answers?

AI search systems and LLMs select passages through a retrieval-then-synthesis pipeline that sits on top of traditional web ranking. Google AI Overviews, ChatGPT with web browsing, Perplexity, and Gemini all introduce a third stage: they retrieve passages from the web, score them for confidence, and assemble a grounded AI answer. Not a replacement for search. It’s an interpretation and synthesis layer that uses search results as grounding data.

The distinction matters technically. Semantic search is a retrieval method: finding content based on meaning rather than exact terms. RAG (Retrieval-Augmented Generation) is a system architecture: it uses retrieval (including semantic search) as input to a large language model (LLM) that then generates a synthesised answer. Lewis et al. (2020) formalised the concept as an architecture that augments language models with a retrieval component to ground generation in external knowledge.

Every AI search product shipping today is fundamentally a RAG implementation. As the Vertex AI Search diagram shows: “Search results” and “Grounded AI Answer” are two separate output surfaces from the same pipeline. Both depend on the same information retrieval and ranking stages upstream.

The strategic question isn’t whether AI search matters, but how it changes the economics of organic visibility.

How do AI systems retrieve information through fan-out queries?

AI systems retrieve information by decomposing the user’s query into multiple fan-out queries, sometimes 10 to 30 sub-queries, each targeting a different facet of the question. Each sub-query triggers its own retrieval pass, generating a broad candidate set of 100 to 300 passages.

These passages are then evaluated through confidence-weighted arbitration. The system scores each passage across four dimensions: relevance to the sub-query, factual consistency with the LLM’s parametric knowledge, entity alignment with the Knowledge Graph, and source authority signals. Only 4 to 5 passages survive this arbitration to form the basis of the grounded AI answer.

The unit of competition in AI search is the passage, not the page.

And the unit of visibility is the citation, not the click. This shifts the ROI model. In traditional search, you measure clicks and sessions. In AI search, the value increasingly comes from being cited as a source, which builds brand recognition and authority even when the user doesn’t click through. The metric shifts from clicks in the browser to clicks in minds.

What makes content citable by AI search systems?

Three properties make content citable by AI search systems: passage clarity, entity accuracy, and structural coherence. Each contributes to the confidence level an AI system assigns to a passage before deciding whether to cite it or skip it.

Passage clarity is the primary arbitration signal. A passage that makes a clear, self-contained claim with explicit entity relationships and bounded scope is far more likely to survive confidence scoring than a passage that requires surrounding context to make sense. If the LLM has to guess what your paragraph means, it will cite someone else’s paragraph instead.

Entity accuracy matters because AI systems cross-reference claims against their Knowledge Graph. If your content makes a claim that conflicts with established entity relationships, or if it is ambiguous about which entity it discusses, the confidence level drops and another source gets cited. Especially relevant for brands that share names with other entities or operate in categories where terminology is contested.

Structural coherence at the passage level means each major section of your content should function as an independent retrieval unit. Each block of 150 to 300 words should contain a declarative claim, define the primary entity, provide supporting evidence, and close the semantic loop. Think of it this way: if someone extracted one section from your article and showed it to a reader with no context, would it still make a clear, complete point? That is the standard AI arbitration sets.

Why are AI hallucinations a retrieval quality problem?

AI hallucinations are a retrieval quality problem because when the retrieval step doesn’t surface high-confidence passages for a facet of the user’s query, the LLM fills the gap from its parametric memory: the statistical patterns learned during training. Parametric memory doesn’t distinguish fact from plausible-sounding fiction.

Current benchmarks place hallucination rates at roughly 3 to 8.5% depending on model and task complexity (GPT-4 benchmarks around 3%; other models higher). Low enough to build user trust, high enough to make retrieval quality the critical variable.

Being the most reliable information source on your topic doesn’t just earn citations. It reduces the probability that AI systems hallucinate about your category, which would protect your brand positioning in AI-generated answers. If no authoritative passage exists for a sub-query about your product category, the LLM will make something up. It will sound plausible. It might be wrong. And your prospects will read it as fact.

What is multimodal retrieval and why do AI agents matter?

Multimodal retrieval is the expansion of AI retrieval beyond text to include images, video, and structured data as retrieval candidates. Google’s AI systems already evaluate multiple content formats. Google AI Mode introduces follow-up sessions where the AI re-retrieves based on the user’s conversational refinements, each follow-up generating new query fan-out against the same and adjacent retrieval pools.

AI agents, autonomous systems that handle knowledge discovery and execute multi-step research tasks, represent the next retrieval surface. The scale of AI retrieval is already significant: AI Overviews reach an estimated 2 billion users monthly and appear in roughly 21 to 55% of Google searches, depending on query category and market (Google announcements; BrightEdge and Ahrefs tracking, 2025–2026).

When an agent searches for “best enterprise deployment tools,” it doesn’t browse SERPs. It retrieves passages, cross-references retrieved information against multiple sources, and synthesises recommendations without human click behaviour entering the loop. Being structurally retrievable for agent queries may become a competitive dimension that outweighs traditional web ranking within a few years.

How does Google search compare to AI-native answer engines?

Dimension	Traditional Google search	AI-native answer engines
Unit of retrieval	Page	Passage
Output format	Ranked list of links	Grounded AI answer with citations
Success metric	Click-through rate, position	Citation, source attribution
Arbitration method	Re-ranking via scoring models	Confidence-weighted passage selection
User behaviour	Click → visit → evaluate	Read answer → maybe click source
Content requirement	Page-level relevance	Passage-level self-containment

And the strategic mindset shift that follows from understanding Google’s pipeline:

Traditional SEO thinking	Retrieval-first strategy
“Rank for target keywords”	“Be retrieved for target query classes”
“Build links to improve authority”	“Build entity-level clarity to expand retrieval surface”
“Create content for keyword coverage”	“Structure content as retrievable knowledge units”
“Measure rankings and traffic”	“Measure retrieval coverage and citation rate”
“Optimise pages”	“Engineer retrieval eligibility across the network”
“Win positions”	“Win the candidate set”

How did Google’s ranking systems evolve from PageRank to NLP transformers?

Google’s ranking systems evolved through six major architectural shifts, each adding a layer that still runs today. Search has always been a core computer science problem. Knowing what each layer does, and when it was introduced, helps you understand which parts of the pipeline your strategy is (or is not) addressing. It also explains why SEO advice from 2015 can be actively counterproductive in 2026.

PageRank era (1998–2010). Google’s original insight was that links between pages function as votes of confidence. Ranking was dominated by link graph analysis. Retrieval was almost entirely lexical, just inverted index lookups with basic term matching. SEO during this period was, for practical purposes, a link-counting game. I started in SEO toward the end of this era, and the transition to what came next caught a lot of practitioners off guard. One day you could rank a thin page with enough links. The next, you couldn’t. Well, not exactly overnight, but it felt that way.

Machine-learned ranking (2010–2015). Google replaced hand-tuned ranking functions with ML models that weight hundreds of signals at once. SEO started becoming a signal optimisation problem rather than a link-counting exercise. The gap between tactical and strategic SEO started widening here. Most agencies kept running the old playbook (more links, more keywords) while Google’s ranking systems were already learning patterns across variables that no human could manually optimise.

RankBrain (2015). Google’s first neural network for query understanding. RankBrain mapped ambiguous queries into vector space, the first step toward neural retrieval. Queries the system had never seen before could now match relevant documents through semantic similarity. For practitioners, this was the moment when “optimise for keywords” started losing its edge over “cover the topic comprehensively.” Many didn’t notice the shift for years.

BERT and NLP transformers (2019). Transformer-based language understanding applied to ranking. BERT enabled contextual query-document evaluation: resolving ambiguities, understanding negation, interpreting prepositions. Google search ranking became truly semantic, not just statistical. After BERT, keyword stuffing went from being a low-value tactic to actively counterproductive in many query classes. The page that used “best SEO consultant” fourteen times would now lose to the page that covered the actual decision journey a buyer goes through.

MUM (2021). Google’s Multitask Unified Model (MUM), announced at Google I/O 2021, expanded the system’s capabilities beyond text. Built on the T5 framework and described as 1,000 times more powerful than BERT, MUM can understand and generate language across 75 languages and process information from multiple formats (text, images, video). For complex queries that previously required multiple searches, MUM could synthesise answers from multilingual and multimodal sources. It represents the bridge between single-format NLP understanding and the multimodal AI systems that followed.

Passage indexing (2021). Google began ranking individual passages within pages independently. A single well-structured paragraph could be retrieved for queries the page as a whole didn’t target. The unit of relevance shifted from page to passage. This change would later make AI-driven answer engines possible, because passage-level retrieval is exactly what RAG architectures need to ground their answers.

RAG and AI Overviews (2023–present). Retrieval-Augmented Generation introduced AI-driven answer synthesis. Search results became grounding data for large language models. As Search Engine Land reported on Jeff Dean’s framing, AI search builds on classic ranking and retrieval rather than replacing it. The system now retrieves passages, scores confidence, and assembles grounded AI answers, adding an interpretation layer on top of the traditional pipeline. As the graph confirms, “Grounded AI Answer” sits alongside “Search results” as a separate surface.

Each era added a layer. Today’s system runs PageRank-derived link signals, ML ranking, NLP re-ranking, passage indexing, and AI synthesis all at once. Strategy that accounts for only one layer optimises for a fraction of the system. I still meet companies whose entire SEO programme is built around the 2010 playbook. Links and keywords. That isn’t wrong, exactly. But it’s incomplete in a way that costs them queries they should be winning.

How does Google classify intent and expand the user’s query before retrieval?

Google classifies intent and expands the user’s query through a query rewrite stage that runs before information retrieval begins. The graph shows this as a distinct step: “Query rewrite” sits between the user’s search query and the Retrieval box. Search engines don’t receive queries passively. They interpret, classify, expand, and augment them before any candidates selection takes place.

Intent classification determines which content formats are eligible for retrieval. A navigational query retrieves brand pages (or, as the graph shows, may bypass the pipeline entirely and route to “Personalized browse”). An informational query retrieves in-depth articles or news articles. A transactional query retrieves product pages and comparison content. Freshness-sensitive queries require timely information and may prioritise recently published or updated sources. If your content format doesn’t match the inferred intent class, it gets filtered out at the retrieval stage, not demoted in ranking. Query-meaning-intent matching at the system level, happening before your content ever competes.

Query augmentation and query expansion broaden the original query into semantically related variations. Google uses synonym expansion, entity resolution, and concept mapping to rewrite what the user typed into what the user likely meant. AI search systems take this further with query fan-out: 10 to 30 sub-queries per complex question, each triggering its own retrieval pass. Your content competes not just for the literal query typed, but for dozens of variations Google generates internally.

Keyword-centric SEO has shrinking returns at scale for exactly this reason. The system isn’t matching keywords. It’s matching intent patterns, entity relationships, and semantic neighbourhoods.

A B2B SaaS company that structures content around the complete decision journey (problem awareness, solution evaluation, vendor comparison, implementation planning) would be eligible for far more queries than one targeting individual keyword strings.

Ever seen a single well-architected page rank for hundreds of queries? That’s what retrieval coverage looks like when query expansion works in your favour. The page didn’t target those queries. Google matched them because the semantic coverage was deep enough to satisfy the rewritten variants.

How should Google’s retrieval pipeline change your SEO budget?

Google’s retrieval pipeline should change your SEO budget by forcing you to diagnose which stage is your binding constraint before you spend another euro. Most SEO strategies are implicitly optimised for ranking: keyword targeting, backlink acquisition, on-page signals. These matter only for content that has already been retrieved.

How to diagnose whether retrieval or ranking is your binding constraint

The diagnostic starts with one question: is the problem that Google cannot find your content, or that Google finds it and ranks it poorly? Detailed analysis of your GSC data will tell you. These require entirely different interventions, and most companies never ask the question.

If you are putting 80% of your SEO budget into ranking work (links, new content production, on-page tweaks) and 20% into technical and structural work (which includes semantic SEO), you might have the ratio backwards, at least for a while. The marginal return on ranking investment approaches zero when the retrieval constraint is binding. Start with foundations, then build new floors.

A structured diagnostic:

1. Audit retrieval eligibility. Are your target pages actually appearing in candidate sets? Google Search Console’s performance data, specifically impressions at zero or near-zero clicks, can indicate retrieval vs ranking problems. A page with zero impressions for a target query has a retrieval issue. A page with impressions but no clicks has a ranking or SERP presentation issue.
2. Assess semantic cost. How much computational effort does the system need to understand what your page is about? High semantic cost, caused by vague entity references, missing structured data, poor heading hierarchy, or orphaned content, reduces retrieval probability. A data science mindset helps here: you are diagnosing system behaviour from observable signals, not guessing.
3. Evaluate passage-level quality for AI eligibility. Are your key claims structured as self-contained, citable passages? For B2B SaaS companies competing for consideration in AI-generated comparisons and recommendations, this is where deals begin, before the prospect ever visits your site.

Why is retrieval eligibility a competitive moat?

Retrieval eligibility is a competitive moat because most competitors in most verticals are optimising for the same ranking signals: building links to the same anchor terms, targeting the same keywords, producing structurally similar content. The SERP is a local equilibrium. Incremental investment in the same signals yields diminishing returns. Sound familiar?

Retrieval eligibility is a different axis of competition entirely. When you reduce semantic cost, improve entity-level clarity, and structure content for passage-level retrieval, you’re not competing on the same dimension. You are expanding the number of queries your content is eligible for, including AI-generated query variations you never explicitly targeted.

In Porter’s terms, this is differentiation rather than cost leadership on the same competitive dimension. The compound returns on architectural investment (topical maps, entity resolution, semantic structure) might well exceed the returns on incremental link acquisition, ceteris paribus. Architecture could scale indefinitely. The link-by-link approach doesn’t.

How does semantic SEO reduce the cost of information retrieval?

Semantic SEO reduces the cost of information retrieval by structuring content around entities, relationships, and meaning rather than keywords alone. It makes content machine-readable at every level, serving both BM25’s lexical matching and neural retrieval’s vector similarity in parallel. The lower the semantic cost for Google’s pipeline to understand what a page is about, the higher the probability that the page enters the candidate set.

A well-designed Semantic Content Network (SCN), what most practitioners call a topical map (though I would argue the term undersells the architecture), creates a closed-loop system where every page reinforces entity relationships through internal links with descriptive anchors. The cluster structure signals to retrieval systems that the domain covers a topic comprehensively, not superficially.

Pillar pages are the retrieval entry points for broad queries. Hub pages organise sub-topics and create semantic bridges between related clusters. Spoke pages capture the long-tail and specific intent variations. Each new page strengthens the retrieval probability of every connected page in the network.

This is the network effect in semantic architecture. And it’s the mechanism behind what the industry loosely calls “topical authority,” though “retrieval coverage” would be more precise. I prefer the second term because it points to what the system actually measures, not what SEOs assume it measures.

Where should you invest next across Google’s search pipeline?

If you are a founder, CMO, or growth lead evaluating your organic visibility investment, the question Google’s pipeline answers is not “how do search engines work?” It is “where should the next euro go?”

Information retrieval is the qualifying round. If your content is not technically accessible, semantically clear, and entity-resolved, you’re not in the competition. No amount of link building or content production compensates for retrieval failure. Diagnosing this is the first step, and it is where most companies never look.

Web ranking is the competition. Once retrieved, your content competes on the strength of user satisfaction signals, authority, freshness, and contextual relevance. But even winning that competition delivers less than it used to. Position #1 historically commanded roughly 40% of clicks, but on queries where AI features appear, CTR may be compressing toward 19 to 26% (First Page Sage, 2025–2026 meta-analysis). The value could be migrating from ranking position to retrieval-plus-citation eligibility.

AI citation is the emerging distribution channel. AI search systems are increasingly where your prospects find solutions and form shortlists, before they ever visit your website. Being citable by AI requires passage-level clarity, entity accuracy, and source authority.

Architecture compounds, tactics do not. Individual page work delivers linear returns. Semantic architecture (topical maps, entity resolution, internal linking systems) delivers compound returns because each element could strengthen the retrieval eligibility of the entire network. For scaling companies, this is the difference between an SEO cost centre and an organic growth engine.

SEO is not a ranking exercise. It is structured influence over a probabilistic retrieval and ranking system, in service of business growth. Google’s pipeline is the territory. Understanding it changes how you invest.

---

The diagnostic I run in every strategic engagement starts exactly here: mapping where the binding constraint sits across information retrieval, web ranking, and AI eligibility, then reallocating investment accordingly.

If that is the conversation you need to have about your organic growth, that is what I do.

---

Powodzenia 🙂

Key definitions

Information retrieval: candidates selection. The process of selecting documents from the index that could satisfy a user’s query, using lexical matching (inverted index + BM25) and neural matching (vector embeddings + ANN search). Precedes ranking in Google’s pipeline.

Ranking: candidate ordering. Machine-learned scoring systems, including NLP re-ranking via BERT and transformer models, that evaluate query-document relevance and assign positions to retrieved documents.

Retrieval eligibility: the structural, semantic, and technical conditions that determine whether content enters the candidate set before ranking begins.

Query rewrite: the stage in Google’s pipeline where the system interprets, classifies, and expands the user’s query before information retrieval begins. Includes intent classification, query expansion, query augmentation, and entity type classification.

BM25: a probabilistic scoring function used in lexical retrieval that weights term frequency, document length, and inverse document frequency to score candidate relevance. Developed from the Okapi weighting scheme by Robertson et al.

Passage indexing: Google’s capability to rank individual passages within a page independently of the page’s overall topic, treating each paragraph-level block as a separate retrieval candidate.

Retrieval-Augmented Generation (RAG): a system architecture (Lewis et al., 2020) that augments LLM generation with a retrieval component, grounding AI-generated answers in externally retrieved content rather than relying solely on parametric knowledge.

Grounded AI answer: the output of a RAG system where the LLM’s response is grounded in retrieved passages and cited sources, as distinct from traditional web ranking results. The term matches Google’s own Vertex AI Search architecture.

Semantic cost: the computational effort a retrieval system expends to understand what a page is about. Lower semantic cost increases retrieval probability.

Entity-level clarity: the degree to which a page’s content unambiguously identifies and relates the entities it discusses, enabling the retrieval system to associate the page with the correct Knowledge Graph entries and query contexts.

Confidence level: the score an AI system assigns to a retrieved passage based on relevance, factual consistency, entity alignment, and source authority. Passages with higher confidence levels are more likely to be cited in grounded AI answers.

Arbitration: passage selection. The process by which AI search systems score individual passages for factual confidence, entity alignment, and source authority before assembling a grounded AI answer.

Fan-out queries: the sub-queries an AI system generates when decomposing a complex question, each targeting a different facet and triggering its own retrieval pass. Also called query fan-out.

Semantic Content Network (SCN): a hierarchical content architecture of pillar, hub, and spoke pages structured around entities and relationships to maximise retrieval coverage across a topic.

Navboost: Google’s behavioural ranking system that uses aggregated user interactions (click patterns, dwell time, navigation behaviour) to adjust web ranking based on real user satisfaction signals, as confirmed in DOJ antitrust trial documents.

Semantic Retrieval Optimization (SRO): an approach for structuring content to optimise retrieval probability across both traditional search and AI search systems (based on Koray Tuğberk Gübür and Sergey Lucktinov’s work). For further reading, see Lucktinov’s published work on the methodology.

Frequently asked questions

Sources:

• https://developers.google.com/search/docs/crawling-indexing
• https://searchengineland.com/google-jeff-dean-ai-search-classic-ranking-retrieval-469386
• https://docs.cloud.google.com/generative-ai-app-builder/docs/ranking-overview
• https://arxiv.org/html/2407.21022v1
• https://sitebulb.com/hints/performance/avoid-excessive-dom-size
• https://law.justia.com/cases/federal/district-courts/district-of-columbia/dcdce/1:2020cv03010/223205/1436
• https://www.justice.gov/atr/us-and-plaintiff-states-v-google-llc-2020-remedies-hearing-exhibits
• https://www.demandsage.com/google-search-statistics/
• https://blog.google/products/search/search-language-understanding-bert/
• https://blog.google/products/search/generative-ai-google-search-may-2024/
• https://ahrefs.com/blog/ai-overviews-study/
• https://arxiv.org/abs/2311.09000
• https://firstpagesage.com/reports/google-click-through-rates-ctrs-by-ranking-position/
• https://patents.google.com/patent/US9405794B2/en
• https://patents.google.com/patent/US7346839B2/en
• https://arxiv.org/abs/2510.05396
• https://research.google/pubs/scalable-in-context-ranking-with-generative-models/
• https://research.google/blog/muvera-making-multi-vector-retrieval-as-fast-as-single-vector-search/