How to measure AI visibility – my AI audit framework

To measure AI visibility, run a fixed list of your customers’ questions many times, from clean sessions with no account history, on every surface they use, and read every mention rate with its margin of error. My AI audit framework asks three separate questions: what models know about a brand, whether they mention it unprompted, and which sources they cite. I’m Szymon Słowik, and I tested it against a commercial tracker.

TL;DR

• A score only means something inside the setup that produced it. Two independently built setups agreed on 27 of 29 threshold calls about the same brand and still differed by 5.7 points per question.
• A single reading of a mention rate is one draw, never the underlying rate itself. Same model, same day, same question: the brand was named in one answer and missing from the next.
• Google’s AI surfaces reach more people than any chatbot. Measuring ChatGPT alone and calling it AI visibility skips the surface most of your customers actually use.
• Search on and search off are two measurements, not one. On the same assistant, turning web search on moved brand mentions by 8.1 and 11.9 percentage points.
• Most brands should not build this. A documented tracker is enough for direction and ordering, at the price of a subscription instead of a research project.

What AI visibility means, and why one score can’t carry it

AI visibility is how often, and how accurately, a brand shows up in the answers AI search engines and assistants generate for questions in its category. Unlike a search ranking, there is nothing fixed to read off a dashboard. Every answer is generated fresh, so any single score is one draw from a distribution.

The comparison later in this article is one anonymised brand, and the rules around it are my working conclusions from client work rather than a market benchmark. Any number out of it supports a claim of one narrow kind: what the models said about that brand, inside the setup it was measured in.

That claim is narrower than most reports make it sound. I think the narrowness is exactly why a board will accept it.

There is no Search Console for ChatGPT, so there are no impressions or positions to attribute from. That changes what the number can support. It tells you about exposure inside a setup you defined, and it says nothing about pipeline or revenue.

Three limits are worth stating before any of the method below.

• It does not measure the effect of optimisation work. My review window for that is eight to twelve weeks, so a five-week measurement shows a baseline and its stability, not growth.
• It does not measure customer behaviour. It measures model responses as a proxy for what a customer asking the same question would see.
• It does not replace a classic SEO audit. Grounded answers depend on retrieval, which is the subject of how AI search differs from classic SEO.

Six mistakes that make AI visibility numbers meaningless

Six mistakes turn an AI visibility number into noise, and I see all six in the wild, agencies included. The table below names each one, what it corrupts, and where this article fixes it.

The mistake	What it corrupts	The fix, and where it lives below
Checking yourself in your own ChatGPT	Your account’s history and memory personalise the answer. You measure what the model knows about you. A stranger gets something else.	Clean sessions, no account history, a fixed location. Step 3 of the procedure.
Not knowing which version or tier you are on	Free, paid and API tiers can route to different models. Two people “checking ChatGPT” may be checking two systems.	Log the model version on every run and pin it where the API allows. Step 3.
Assuming AI means ChatGPT	Google’s AI Overviews and AI Mode sit inside Search and behave differently from the chat assistants. In Tomek Rudzki’s 500,000-prompt sample for Peec AI, AI Overviews appeared 86% of the time in a sample skewed towards business and buying prompts.	Surfaces chosen by who your customer is, measured and reported separately. Step 4.
Treating a one-shot prompt like a search query	A query hits an index. A prompt starts a conversation and, in search mode, fires its own sub-queries. Keyword-shaped text typed into a chat returns a definition in which no brand needs to appear.	Keyword-shaped queries on Google surfaces, question-shaped prompts in chat. The prompts section.
Checking once per round	Each answer is a fresh draw from a probability distribution. One reading reports that single draw as if it were the underlying rate.	Twelve repetitions per prompt, per surface, per round, every rate reported with its interval. Step 5.
Forgetting how AI conversations work	A number with no question behind it. Knowing a brand, mentioning it and citing it are three different things.	An audit built on exactly those three questions. The audit section below.

Ever compared two screenshots of the same question and wondered why they disagree? The fifth mistake is the one that quietly breaks everything else. An answer is a draw from a distribution, so the screenshot you took this morning is not a fact about the model, it is one sample you cannot reproduce on demand. Everything below about repetition exists because of that single property.

How to measure AI visibility across AI search engines, step by step

To measure AI visibility, build a prompt list from real demand and freeze it, split unaided prompts from aided ones, then run every prompt repeatedly on each platform your customers use. Report each platform on its own, with a margin of error on every rate. The eight steps below are the sequence I run.

1. Build the prompt battery from documented demand, then freeze and version it.
2. Split unaided prompts from aided ones.
3. Ask real questions from a fresh, clean session, and log the setup.
4. Choose surfaces by your customers, and keep them separate.
5. Fix a weekday, then repeat every prompt the same number of times.
6. Record every rate with its interval and its failures.
7. Tag every mention, collect sources, take competitors from the answers.
8. Read at the aggregate, and report per platform and product line.

Step	The rule it enforces	What you end up with
1. Battery	The battery comes from documented demand, never a whiteboard. A material change to the list starts a new series.	A prompt list with demand behind every line
2. Split	No brand name and no steering in unaided prompts. Branded and forced-shortlist prompts live in the aided set.	Two readings that diagnose each other
3. Clean position	A real user question, a fresh conversation every run. Log model version, route, language and account state.	An unpersonalised baseline you can repeat
4. Surfaces	Platforms measured and reported separately, never averaged. Write the roll-up rule before you start.	The right denominator for each platform
5. Repetitions	Never a single run. Choose and document the repetition count from the precision you need and the budget you have. This engagement used twelve.	Within-round variation you can actually estimate
6. Intervals	Decide before you start whether a failed or empty answer counts as no mention or is dropped. Report the interval method you used on every mention rate.	A margin of error you can defend
7. Tags and sources	Competitors come from the answers during the pilot, then the set is frozen. Strategic rivals are tested separately and labelled.	Know, mention and cite all answered
8. Reading	A scorecard of separate numbers, one row per platform. Compare against your own prior window.	A number that survives scrutiny

Step 3 is where most homemade studies go wrong, and not only through personalisation.

I tested whether a “stabilising” instruction attached to each prompt changes the result. Answers came back just as substantive without it and about 35% cheaper to run, on a small test of two prompts, so treat that as a direction.

Leave your own instructions out for a second reason. In one pilot the model recommended a provider that does not exist, and an anti-hallucination instruction would have suppressed exactly that answer. A model inventing a competitor is a finding worth keeping.

Step 7 is where the surprises live.

On one engagement the answers kept naming an app-based competitor that nobody in the room had on their list. No tool configured with the brand list you walked in with will ever show you that.

Where AI visibility prompts should come from

AI visibility prompts should come from documented demand: your own search data first, then your customers’ questions, then category research for the questions your site doesn’t reach yet. On the engagement below, a twelve-month Search Console export of 2,687 rows reduced to 15 queries with confirmed traffic, alongside 60 research questions and 33 search-type queries measured separately.

Where the phrases came from	Proposed	Confirmed demand	No recorded volume
Search Console export	15	15 (40 to 4,500 searches a month)	0
Client’s intuition	7	4	3 (not in the keyword index at all)
Client’s hypothesis for one product line	15	5	10

Ten of the client’s fifteen suggested phrases for one product line had no recorded search volume. That meeting was not comfortable, and I remember it because the client was not wrong. The demand existed, under different wording than they assumed, so those exact phrases did not go in.

Which is why the rule exists. No recorded volume is not proof that nobody asks, it is a reason to check the wording against real customer questions before it enters a battery.

A prompt is not a keyword, so the two do not belong on the same surface. Keyword-shaped queries are measured on Google surfaces, question-shaped prompts go to the chat assistants. Type “crm software” into a chat and you get a definition in which no brand needs to appear.

Measuring that produces a number with nothing in it.

There is a mechanical reason behind the rule. An assistant in search mode fans one prompt out into several queries of its own. Tomek Rudzki at Peec AI measured 2.1 on average across five million fan-outs.

Your prompt list is therefore a versioned list of intents, grouped by what the customer wants. Exact strings will not survive the fan-out.

How many prompts you need is the same question as how many repetitions, and it has the same shape of answer. Count the intents your customers actually have, multiply by the product lines you sell, and let that set the number. The engagement below ran 43 indicator questions, which was what those intents came to rather than a figure anyone picked.

Language and country belong in the setup, not in a footnote. A brand selling into three language markets is running three batteries, because a question asked in one language gets answered from a different set of sources.

Log the locale next to the model version on every run.

Splitting by city, on the other hand, bought nothing. I tested it. The brands that came up did vary between cities, but randomly rather than regionally, and two cities disagreed on cited sources about as much as two different days in one city did. So the money goes into repeating from one national location, which neutralises the same variation without mixing two sources of it.

What an AI visibility audit measures: know, mention and cite

An AI visibility audit answers three questions about a brand: what the models know about it, whether they mention it unprompted, and which of its pages they cite. Those are three different tests, and a brand can pass one and fail the other two. Some people call the same exercise an LLM visibility audit.

What the three tests produce is read through four groups: exposure, competition, answer quality and visible sources. One decision comes before all of them, and it is which legal entities, sub-brands and product names count as “you”. Settle that before the first run and never redrawn because a result came back awkward.

How you match the name matters as much as which names you picked. Searching the answer text for the brand as a plain string counts every time those letters sit inside another word. On one engagement, counting that way against proper entity extraction moved the mention rate by 18.7 percentage points on a short brand token. That is the matching rule talking, not the market.

Layer	Which question it answers	What it measures	The false positive it catches
Unaided mention rate	Mention	Share of open answers that name you at all	Scores inflated by prompts that name the brand
Aided share of voice	Mention	Your share of a forced shortlist, the brand still unnamed in the prompt	Reading a strong shortlist position as strong recall
Position and role	Mention	Leader, alternative or footnote, when you appear	Counting a footnote as a recommendation
Class	Mention	Which kind of provider the models recommend: incumbents, specialists, app-based challengers. Counted by whom the customer would actually sign with	Blaming the brand when the models skip the whole category
Head-to-head win rate	Know	Whom the model picks when asked “you or them”	Assuming the model knows your positioning
Sentiment and arguments	Know	Why you are named, tagged from a fixed dictionary	A brand named as one to avoid counted as a win
Factual correctness and hallucination	Know	Whether what the model says about you is true	Recommendations of products that do not exist
Sources	Cite	Which domains grounded answers draw on, and which of them mention you	Being cited on pages that recommend the competitor

What AI models know about your brand

What a model knows, for the purpose of an audit, is what it says about your brand when you ask about the brand directly, and correctness and sentiment are scored on every one of those mentions. A brand named as the one to avoid is not a success, and counting mentions alone would record it as one.

There is a failure mode here that has nothing to do with your marketing. Jan Ehrlinspiel at Peec AI tested 904 brand names. Ordinary dictionary words used as brand names were misread as the common noun by roughly one model in twelve, with no equivalent failure in the well-known-brand group.

How brand mentions and share of voice differ

Brand mentions split into two numbers that have to be read together: how often the model brings you up unprompted, and what share of a forced shortlist you take when it has to build one. The first is the unaided mention rate, which is answers naming you unprompted divided by all answers for that prompt set, surface and window.

Who the models name beside you is a separate reading, and a useful one. When the brand appears, I record the other brands in the same answer and report it as a plain conditional: when the models name you, how often do they also name each rival.

Deliberately not a lift score. Lift divides by how often the rival appears overall, which blows up into meaningless numbers exactly when your own brand is rare, and rare is when you most need the reading.

Low unaided mentions alongside a high aided share is a diagnosis in itself. It means the models do not recall you on their own but rate you well once reminded. That is a different problem, with a different fix, from being rated badly.

Which sources AI engines cite

Grounded answers show some of their sources, and those cited domains are a partial map of where the model reads about your category. Ungrounded answers show none. Collect every cited domain, then look hard at the pages cited most often that never mention you.

One case from my own client work. A meal-prep subscription brand had better ratings and reviews than its competitors, and almost none of that was visible anywhere except its own site. An advantage that lives only on your own pages rarely travels into generated answers.

We had it stated and evidenced on around 50 third-party blogs. The brand now comes up across the answers for the superlative prompts in its category, the local-language equivalents of “best”, “recommended” and “most often chosen”.

I never ran a controlled before-and-after on it, so treat it as a case rather than a study. It is one I am certain of, and one I have walked through on conference stages.

Michał Suski, co-founder of Surfer, found in a 90-day sample that the authority metrics he tested did not predict how often a page was cited. Surfer’s tracker appears later in this article behind an affiliate link, so take that as his finding, not my endorsement.

Structure on your own pages moves the same needle. Waikay added one structured-data layer to its own B2B SaaS site and watched Bing Copilot citations over 19 weeks with no new content and no new links: an average of 167 citations a week before, 614 after. One vendor measuring its own domain is a direction worth testing, not a benchmark.

For every cited source I also tag what probably won it the citation: concrete numbers, freshness, credibility signals, the size of the comparison, brand strength, or simply the type of material. That turns a list of domains into a list of editorial instructions.

My working model is that a citation is a passage retrieved rather than a page read, and it has some support. Metehan Yesilyurt at Peec AI read the undocumented behind-the-scenes log of one prompt and saw 223 pages fetched, 16 cited, and content split into blocks of roughly 170 words.

One researcher reading an undocumented log is support for a model, not proof of it. Either way a page gets cited for one passage. That passage has to answer the question on its own.

The plain version: a page has to be retrievable and eligible before any ranking signal can get it cited. I covered that in off-site SEO in the age of AI.

The score moved. Is that real?

Usually not by itself. The same model, on the same day, with the same question, has named a brand in one answer and left it out of the next. One mention across two runs puts the true mention rate anywhere between 9% and 91%.

Repetitions per prompt	Interval half-width	What each extra run buys at this point
5	plus or minus 33 pp
8	plus or minus 28 pp	1.5 pp per run
12	plus or minus 25 pp	1.0 pp per run
20	plus or minus 20 pp	0.6 pp per run
30	plus or minus 17 pp	0.3 pp per run
50	plus or minus 13 pp	0.2 pp per run
100	plus or minus 10 pp	0.07 pp per run

Those widths are per single prompt, which is exactly why a single prompt is never the unit of reading. A whole product line runs to several hundred observations a week, and the margin there comes down to around 9 points. Cumulated over five weeks it is narrower again.

So the report says the brand sits at some level in this product line, plus or minus 9, and never says what happened in question 14 in week 3. Single-prompt resolution is not something I promise, and that is an honest limit of this kind of measurement rather than a flaw in the setup.

Those are 95% Wilson intervals, the margin-of-error method used here, assuming independent runs. They cover the randomness of the answers. They do not cover whether your prompts are the right ones, or changes to the model and to the pages it reads.

Ever watched a dashboard move three points in a week and wondered whether anything actually happened?

Twelve repetitions per prompt, per chat surface, per weekly round was this engagement’s protocol, chosen from the precision needed against the budget available. The Google surfaces took five, because an AI Overview is not built fresh for every request and settles down with fewer reads. It is not a minimum for yours. What the repetition buys is a measurement of the variation itself.

Fixing one weekday separates two things that otherwise blur. Same-day spread across repeated runs is what the system does on its own. Week-to-week change is everything else that moved, including the pages the model reads, which is not the same thing as model drift.

A change counts when this week’s mean falls outside last week’s interval, read across a whole product line rather than at a single prompt. Anything smaller than that gets called model noise out loud, in the report. Across dozens of prompts and several surfaces, a few will cross a threshold in any given week by chance alone.

What is a good AI visibility score?

A good AI visibility score is one that beats your own earlier reading by more than the sampling noise, or beats a competitor measured in the exact same setup. No score is good on its own, because it is a property of the setup that produced it.

Scores do not travel between tools because the denominators, the weights and the entity rules all differ. None of that is a bug in anyone’s product. What you are seeing is what happens when two teams define the same word differently and both report a number to two decimal places.

Google’s own AI Overview for this question quotes vendor bands, above 50 is good and above 70 is dominant, and then says there is no universal benchmark, in the same answer. The bands trace back to vendor blogs with no disclosed validation on the pages linked. So I don’t use them.

Four rules make a score readable.

1. Compare only readings from the same setup version.
2. Never read a mention rate without its sample base and its margin of error.
3. Judge against your own prior window, or a competitor in the same panel, never against another tool’s level.
4. When a share of answers moves, check the raw counts behind that share before calling it a business effect.

The fourth rule has a good worked example. David Konitzny at Peec AI saw listicles’ share of ChatGPT sources halve while their indexed count barely moved, 100 to 92. The share collapsed because the denominator grew. Listicles lost nothing.

What an AI visibility report to the board should contain

Report a scorecard of separate numbers, one row per platform and product line, never one composite score. These are the fields a defensible scorecard needs.

Field	What goes in it
Platform and product line	One row each, never merged
Setup and prompt-set version	The version identifier the reading was produced under
Comparable window	The dates, and the prior window it is being compared against
Prompts times valid runs	The sample base, plus how many runs failed or returned nothing
Current unaided mention rate	The rate with its 95% interval
Change from the prior window	The difference, its interval, and whether it passed the agreed threshold
Aided share of voice	Your share of forced shortlists in the same window
Wrong or negative answer rate	Factual errors and negative mentions, reported separately from each other
Visible-source signal	Which of your domains were cited, or “n/a” on surfaces that show no sources
Decision and owner	What this row changes, and who owns it

One more split sits underneath that table. Inside a product line, questions get tagged core, segment or feature. Otherwise a general question gets averaged with an industry-specific one, and the average describes neither.

The row then reads properly. This line sits at 35%, carried by the core questions at 55% and dragged down by the feature questions at 15%, which is a sentence somebody can act on.

People ask me for the one number to put in front of the board. I do not have one, and I would be careful with anyone who does.

A well-built AI visibility audit is closer to market research than to a rank tracker. The point is not to check AI rankings. It is to learn which sources the models lean on, who gets mentioned and how often, and what brings a brand up at all.

The scorecard is the cover sheet. The deliverable underneath it is the list of missing sources, factual errors and structural gaps.

The interval on each row covers the sampling of repeated outputs under that setup. It does not cover attribution, revenue, or anything else that could be wrong upstream of the measurement.

When a tracker is enough, and when you need to build your own

A documented commercial tracker is enough when the question is whether you show up against a competitor list you already have. It stops being enough the moment the real question is who else is showing up that you never thought to check. The difference is application, not tool quality.

Trackers are genuinely good at the first job. They are comparatively cheap, they run single question-and-answer scenarios against named brands, and some of them also return a long list of cited sources. For checking whether you appear in response to specific prompts, I recommend them without hesitation.

The second job is different work: testing the same answers against criteria you choose, such as a wider library of brands, sub-brands, product names or attributes.

A build is smaller than the word suggests. Underneath, it comes to one call per prompt per surface, somewhere to keep the answers, and a dictionary that tags each one. I run a separate folder and a separate small database per study, because a study that shares storage with the last one stops being reproducible the moment either of them changes.

The hard part is none of that. What actually costs you is deciding what counts as a mention before you see the answers, then not changing your mind once you have.

One practical warning before you automate anything. Reading an interface with a script is not the same permission as using a documented API, so check the terms of whichever route you pick.

Here is the case that made the distinction concrete for me. On one engagement, a direct competitor’s own brand name barely appeared in the answers at all, while a service that same competitor runs under a different name came up repeatedly.

A tracker configured with a list of competitor brand names would have reported that rival as absent. The rival was not absent. It was present under a name nobody had thought to add to the list, and no fixed brand list can surface an entity that isn’t on it.

Option	Use it when	What it takes to run	Minimum evidence to demand	What it cannot claim
Free manual check	You need a rough baseline this week and nothing depends on it	An afternoon to set up, then a few hours a week. It stops being free the moment you want a trend	Clean signed-out sessions, the same questions every time, every answer saved	Any trend, and any comparison between two periods
Documented commercial tracker	You need direction, ordering, and threshold calls against brands you can name	A subscription, plus a few hours a month from someone who reads it properly	Route and model pinning disclosed, repetition count disclosed, raw answers exportable, you own the prompt list, platforms kept separate with stated entity rules	An absolute level comparable to any other tool’s level
Custom build	You need raw answers, pinned models, or open-ended competitor discovery. Board scrutiny on its own is not a reason, because a documented tracker survives that too	Days of setup, then a fixed running cost per round. Five rounds here came to 10,440 answers	Your own logged setup, versioned prompt list, intervals on every rate	Immunity from backend changes you cannot see

The honest version of the cost question is that a build is not expensive in money, it is expensive in attention. The running cost of a round is small next to the time it takes to keep a battery versioned, tag answers the same way every week, and notice when a surface quietly changes underneath you. Budget the person, not the tokens.

On the engagement behind this article I did not actually choose between them. The source radar was built twice over, once from my own measurement and once from the tracker’s own citation collection, and a domain appearing in both was treated as a confirmed target.

A domain that showed up in only one was reported as the weaker signal it is. That is the honest shape of this question, because two independent readings of the same thing are worth more than either alone, and the second one buys you the standing to disbelieve the first.

Two honest limits on the tracker I checked. Surfer does not disclose its repetition count, and its API returns no raw answers, which is exactly why I ran the comparison in the next section rather than assuming agreement.

Disclosure before the link: I use Surfer under a partnership and the link below is an affiliate link. I checked its numbers against my own measurement before recommending it, and that comparison is the next section. It is the only such comparison I have run, so this is not a claim about other tools: Surfer AI Tracker.

And if the honest answer is that you want this run for you rather than built in-house, that is what a strategic SEO audit covers, with the AI visibility work running inside it.

What to ask the agency or tool that sent you a score

Five questions separate a score you can act on from one that only looks like it. Ask them of whoever sent you the number, and treat a missing answer as a reason not to trust it.

1. Which model and which route produced these answers, and were both held constant across the whole period?
2. How many times was each prompt run, and on which days?
3. Can I see the raw answers, or only the score built from them?
4. Whose prompt list is this, and can I change it and keep the history?
5. Which entities count as us, and were the platforms reported separately or averaged?

None of the five is a trick question. Any setup built by someone who took the measurement seriously can answer all of them in a couple of minutes.

What this method can’t tell you

This method has real limits, and the biggest one is time. A five-week measurement window shows a baseline and its stability, not whether anything you changed worked. My review window for optimisation work is eight to twelve weeks, and nothing shorter answers the question “did this help”.

It measures model responses. What a customer then does with them is a separate question nobody has good numbers on yet, so every mention rate in this article is a proxy for exposure.

Here is the distinction I keep coming back to with clients. What you published is an activity measure. Whether the model now serves you when a customer asks is an outcome measure, and the second one is what to report.

It still does not attribute pipeline, revenue or ROI. Those need referral traffic, conversions and CRM data sitting next to it.

Two cheap signals sit next to this one and answer what it cannot. Your analytics will show sessions arriving from the assistants, and your server logs will show whether the AI crawlers fetch your pages at all. Run both alongside the measurement, because together they cost almost nothing.

Neither one tells you what a model says about you. They tell you something this method never will, which is whether anybody arrived and whether your pages were ever read. A page the crawlers never fetch cannot be cited, and that is a retrieval problem rather than a visibility one.

The agreement result in the next section does not make scores portable. Two setups can rank your questions identically and still report levels several points apart, so a number from one tool still cannot be read against a number from another.

Whether two systems track change over time equally well is untested here.

My comparison covers agreement at one point in time, and whether two systems agree about movement is a different test that I want to run next.

And one question in my own data still defeats me. On every pairing I have run, one of the 29 shared questions disagrees between the two systems, and I cannot yet tell which reading is the right one.

What I found when I measured the same brand two ways

In the one comparison I have run, my setup and a commercial tracker agreed on which questions the brand won and lost, and disagreed on the level by 5.7 points per question on average. The rest of this section is how that finding was produced.

The argument about these numbers is live and evidence-free. Seth Besmertnik, co-founder and CEO of Conductor, which sells an AEO platform, has said most tool scores are unreliable while AI search itself is very measurable.

Read what he asks next, because it is the argument of this whole article coming from someone who sells the category. When he hears a visibility score, he asks which persona, which stage of the journey, which product area, and branded or not. That is a man arguing against the convenient metric his own market runs on.

We disagree about the fix. He builds narrow indexes per persona, and I keep one versioned battery and report it split by platform and product line. We agree on the thing that matters, which is that one number across everything hides the answer.

Reddit threads ask how much to trust a vendor’s score. Nobody in that argument brought a measurement, so I ran one.

I ran 43 custom indicator questions for a big corporate brand and restricted the comparison to the 29 that were identical, word for word, in the tracker’s 45-prompt set. The other 16 tracker prompts were left out, three of them because they return software from an unrelated category.

Six surfaces were measured. Google AI Overviews and AI Mode were read from the live result, then ChatGPT and Gemini each with and without search, through the API with models pinned. Those surfaces were chosen for that client’s customers and are not a recommendation for yours.

Two ways to find out which platforms your own customers use: put the question into your next round of customer calls, and check which assistants already send you referral traffic in analytics.

Perplexity, Copilot and Claude are missing from that list on purpose. This client’s customers were micro and small businesses who do not touch them, and measuring a channel your audience is not on lowers the validity of the study rather than making it more complete.

Perplexity data arrived anyway through the tracker, so it sits outside the scorecards as a side observation rather than being thrown away. The same test would put these surfaces into the main measurement for a different audience, which is the point of running the test at all.

My side ran five weekly rounds between 10 August and 7 September 2026, 10,440 answers in total. The tracker’s side was a rolling 90-day window read on 14 September, across 41 daily reports.

Those are different units and I am not pretending otherwise.

Measure	First analysis (31 Aug)	Cumulative update (14 Sep)
Shared questions	29 of 43	29 of 43
Difference between the two pooled headline readings (reference only, pooled to match the tracker’s own headline)	1.9 pp	2.5 pp
Mean absolute difference per question	5.5 pp	5.7 pp
Median absolute difference per question	4.3 pp	3.7 pp
Correlation (Pearson r)	0.86	0.88
Rank correlation (Spearman)	0.78	0.84
Same side of the engagement’s 15% operating cutoff	27 of 29	27 of 29
Questions diverging by more than 10 pp	4 of 29	6 of 29

Read that table in two halves. The levels disagree, by 5.7 points per question on average and by more than 10 points on six of the 29. The ordering agrees, at a rank correlation of 0.84, and 27 of 29 questions landed on the same side of the cutoff that engagement was operating to.

Four things explain the gap, and I can name three of them.

• Entity resolution. The tracker resolves brands as entities, so two sub-brands I counted as the parent were counted separately by it. That explains roughly half of the first run’s gap, in the same direction.
• Sampling design. Twelve runs inside one round estimate within-round variation. One draw per day for 90 days mixes that variation with retrieval changes and silent system changes.
• Route. The tracker reads interfaces and I used the API. Malte Landwehr, measuring 555 prompts across 47 runs per route, found brand visibility differed by 41% between API and interface.
• The remainder. What is left cannot be decomposed without the tracker’s raw answers, which its API does not return. So I am not going to claim I know.

The largest single effect I found sits inside my own data rather than between the two systems. On the same assistants and the same questions, turning web search on moved the brand’s mention rate by 8.1 and 11.9 percentage points against the same assistant with search off.

A tool that merges those two modes into one number is averaging two different things. The user can switch search on, and the product sometimes triggers it without being asked, so neither figure on its own is “visibility in ChatGPT”. Report them apart, or say which one you measured.

One last piece of honesty about this comparison. The September figures add data to the first analysis rather than replicating it independently, and neither system is a ground truth for the other. What the exercise establishes is the shape of the disagreement.

Frequently asked questions about measuring AI visibility

Measurement tells you which of the four audit groups is losing, and on which surface. Changing it is content, structure, authority and off-site work, and that starts after the measuring.

If you want the measurement run and the plan built from it, that is what a consulting engagement is for, and the framework I connect the two halves with is BUXS.

If you have run a comparison like this and got a different answer, I would genuinely like to see it, because two data points beat one. And if your tracker gives you a score to one decimal place with no interval next to it, now you know which question to ask. Pozdrawiam!

The know-how input, insights and conclusions: Szymon Slowik. Research and drafting support: AI-assisted BUXS engine. Reviewed, corrected and finalized: Szymon Slowik.

Created by Szymon, edited with an LLM… see any difference? 🙂