AI visibility tracking tools are accurate enough to show a trend, but no single score is exact, because AI answers change between runs. Whether a tracker is accurate for you depends on how many prompts it asks, how often it repeats them, and whether it keeps brand-name prompts out of the main score.
This page explains where the errors come from and gives you a five-step test you can run on any tracker, including ours, in an afternoon. Visiby sells an AI visibility platform, so we have tried to keep the test fair to every tool.
01 — VarianceWhy do AI answers change between runs?
AI tools write each answer fresh, so the same question can produce a different list of brands and sources a minute later. That is the starting point for every accuracy question.
Three primary sources explain why:
- OpenAI's guide to its seed setting says it makes a "best effort to sample deterministically" and that "Determinism is not guaranteed" (OpenAI Cookbook). Even developers who ask for repeatable answers do not always get them.
- Google says "AI Mode and AI Overviews may use different models and techniques, so the set of responses and links they show will vary" (Google Search Central). Google also says both may use a "query fan-out" technique, running several related searches behind one question.
- A SparkToro study published on 27 January 2026 had 600 volunteers run 12 prompts through three AI tools a combined 2,961 times. There was less than a 1 in 100 chance of getting the same list of brands twice, and about 1 in 1,000 of getting the same list in the same order (SparkToro). Its author still calls how often a brand appears across many runs "a reasonable metric", but not its position in a list.
So one run is a snapshot. If a tracker asks each prompt once and shows the result as a fixed ranking, it is more confident than the answers allow. We look at how much answers move in more detail in AI answer volatility.
02 — What is countedWhat does an AI visibility tracker actually measure?
A tracker asks a fixed list of questions in one or more AI tools and records what comes back. Three different things can be counted, and they are not the same:
- A mention: your brand is named in the answer.
- A citation: one of your pages is linked as a source.
- A recommendation: the answer presents you as a pick.
A tool that counts mentions and one that counts citations can show very different numbers for the same answers, because an answer can name you without linking to you, or link to you without naming you. Before you compare two tools, check which of the three each one counts and in which AI tools. A brand can be visible in ChatGPT and missing from Google AI Overviews, so a single blended "AI search" number hides useful detail.
03 — Sample sizeHow many prompts and runs does a score need?
Enough that one odd answer cannot move the score much. With ten prompts asked once, a single changed answer moves the score by ten points. With many prompts, each asked several times, one changed answer barely registers.
Here is a made-up example. If a brand appears in 4 of 10 answers, the true rate could be anywhere from about 17% to 69%. At 40 of 100 answers, the range narrows to about 31% to 50%. These are 95% ranges from the Wilson method (NIST), and they assume the runs are alike: same wording, same tool, same setup.
Numbers alone are not enough, though. The prompts must match how your buyers ask. A useful set mixes these kinds of question:
| Prompt type | Example | Why it matters |
|---|---|---|
| Category | best tools for tracking AI search visibility | Shows whether you appear when buyers browse |
| Problem | how do I know if ChatGPT recommends my brand | Catches buyers who do not know the category name |
| Use case | AI visibility tracker for a B2B software team | Matches narrower, high-intent questions |
| Comparison | X vs Y for AI search tracking | Shows how you are framed next to rivals |
| Brand | is [your brand] any good | Checks what AI says about you, reported on its own |
A large list of questions your buyers never ask is as misleading as a short one. Ask any vendor how prompts are chosen, grouped and retired.
04 — AI checkWhat did AI tools say when we asked?
We asked the five questions behind this page in four AI tools on 29 October 2026 and logged every answer. Here is what appeared. It shows what the tools said, not why they said it.
Method: one new chat per question, English, run from India. ChatGPT and Perplexity were signed out. Google AI Overviews and AI Mode ran in a browser signed in to a Google account, so personalisation may apply. Gemini was not tested, so 20 of the 25 planned runs have answers.
| Question | Answered | Trackers named in the answer text | Visiby named |
|---|---|---|---|
| Are AI visibility tracking tools accurate? | 4 of 4 | 1 tracker, in 1 answer | 0 of 4 |
| How can I test whether a tracker's numbers are reliable? | 4 of 4 | None | 0 of 4 |
| How many runs per prompt does a score need? | 4 of 4 | 2 trackers, in 1 answer. 3 answers cited pages from tracker vendors as sources | 0 of 4 |
| Why do different tools give different scores? | 4 of 4 | None recommended | 0 of 4 |
| Which trackers show the answers and cited links? | 4 of 4 | 13 different trackers, 5 or 6 in each answer. None was in all 4 answers; the most named was in 3 | 0 of 4 |
All four answers to the first question said a single run is a snapshot and that scores should be read as estimates. On how many runs are enough, the four tools gave four different answers:
| Tool | Runs per prompt it suggested |
|---|---|
| ChatGPT | 10 to 20 to start; 30 to 50 or more for benchmarking |
| Perplexity | 3 to 8 for a direction; about 100 per prompt and engine for a rate within about 10 points |
| Google AI Overviews | 3 to 5 as a floor; about 300 to 400 answers in total for a tight trend |
| Google AI Mode | 12 to 100 |
Visiby was named in 0 of the 20 answered runs.
05 — Brand promptsWhy do brand-named prompts inflate scores?
A question that names your brand almost always gets an answer about your brand. If a tracker mixes those prompts into the main score, the score rises without telling you anything about discovery.
Brand prompts are still useful. They show how AI tools describe you, whether facts are out of date, and which sources they lean on. They belong in their own view, next to the discovery score, not inside it.
When a vendor shows you an impressive number, ask what share of it comes from prompts that already contain your name.
06 — The 5-step testHow do you test any AI visibility tracker?
Run this five-step test before you trust a tracker's numbers. It works on any tool, and it takes an afternoon.
- Read the prompt list. Check that most prompts are questions your buyers ask without naming you, and that brand prompts are reported separately.
- Repeat five prompts by hand. Open each AI tool in a private window, signed out, start a new chat for each question, and note who is named and which links appear.
- Compare your notes with the dashboard for the same day. The tool should let you open the answer text and cited links behind each result, not only a score.
- Ask the same five prompts again a few days later. If the answers moved, check that the tool shows that movement through a run history or a last-seen date.
- Write down the setup: the date, each AI tool and model if shown, signed-in state, country and exact wording. Ask the vendor for the same details about how their answers are collected.
A tracker that passes this test will not remove uncertainty. It will show it to you, so you can tell a pattern from noise.
A tracker does not see exactly what each buyer sees. OpenAI says its search may use a user's saved memories and an approximate location from their IP address when it rewrites a question (OpenAI Help). A signed-out check or a tracker run gives you a fixed, repeatable view, not a copy of every buyer's screen.
07 — A good reportWhat should a trustworthy AI visibility report show?
The score, the evidence behind it, and the limits of the measurement. Look for:
- clear definitions of mention, citation and recommendation
- separate results for each AI tool
- brand prompts split from discovery prompts
- the answer text and cited links behind each result
- run history, so one-off appearances are not read as wins
- what changed since the last report, and which prompts drove it
A score without evidence cannot be checked. Evidence without a score is hard to act on. The useful reports give you both. For a walk-through of each part, see how to read an AI visibility report.
If your number jumps overnight and nothing changed on your site, open the prompts behind it before you act. Common causes are different answers in the latest run, an edited prompt list, a new model or engine, failed answers dropped from the count, and brand prompts added to the main number. Only prompts with the same wording, on the same tool, in both reports show that something really moved.
08 — Where Visiby fitsWhere does Visiby fit?
If you want to measure this yourself, run the free check: it shows what ChatGPT says about your brand for real buyer questions, and you can compare it with your own hand check. For tracking across more tools and weeks, see the AI visibility platform. If you are still choosing between trackers, our list of AI visibility tools sets out what each one covers.
Visiby, our tracker, saves the full answer and the sources each one cites. It counts a mention (your name as a whole word) apart from a citation (a link to your site), and it leaves prompts that contain your own brand name out of its rates (how Visiby counts). It does not track Gemini or Microsoft Copilot, so it is not the right tool if those matter most to you.
09 — FAQFrequently asked questions
10 — SourcesSources
- OpenAI Cookbook, How to make your completions outputs consistent with the new seed parameter (6 November 2023), checked 29 Oct 2026
- Google Search Central, AI features and your website, updated 10 December 2025, checked 29 Oct 2026
- SparkToro, NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility (27 January 2026), checked 29 Oct 2026
- OpenAI Help Center, Searching the web with ChatGPT, checked 29 Oct 2026
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1. Confidence intervals, checked 29 Oct 2026
Virender Singh is the technical lead at Visiby, where he builds the crawling, structured-data, and answer-engine analysis behind the product. He writes about the technical mechanics of answer engine optimization. View full profile →