How should you compare measurement reliability?

Ask suppliers to explain disagreements, not eliminate all variation. Research published in 2026 documents variability across repeated AI-search observations and argues against treating a single result as a stable visibility measure. These papers support repeated measurement, but do not establish a universal sample size for your software category. (arxiv.org)

Broad databases vs buyer-owned prompt panels

Ahrefs explicitly describes the modelling behind its broad database. Peec’s current pricing page describes its prompt-volume measure as a relative topic-demand score from one to five. Do not assume that any number labelled “prompt volume” is a directly observed count of customers asking that exact question. (ahrefs.com)

What should a supplier demonstrate during a common test?

Our proposed acceptance test has six parts:

1. Correct entity recognition. Test abbreviations, product names, parent companies and a similarly named unrelated business.
2. Separate outcomes. Inspect an answer that names the brand but does not recommend it, and one that recommends it without linking an owned page.
3. Evidence recovery. Export the answer, exact prompt, timestamp, surface, location settings, source URLs and scoring result.
4. Failure accounting. Identify blocked requests, timeouts, missing citations and non-triggered AI features without turning them all into zeros.
5. Method-change handling. Show what happens to historical comparisons if the supplier changes collection routes, models or prompts.
6. Human review. Have your analyst review ambiguous recommendations and material factual errors before accepting automated scores.

For ChatGPT checks, include memory and location settings in the collection record because OpenAI documents their possible influence on search. (help.openai.com)

How many prompts and repetitions are enough?

There is no universal number established by the sources reviewed. IAB’s framework calls for repeated responses and a sufficiently broad, disclosed query set; its criteria treat fewer than 50 queries per measurement programme as exploratory rather than category-level directional measurement. That is framework guidance, not a guarantee that 50 makes a benchmark reliable. (iab.com)

For a B2B software pilot, we propose 60 core prompts as an initial design, divided across discovery, requirements, comparisons and shortlisting. This is an editorial starting point—not measured US search demand or a validated minimum.

Run an initial repeatability exercise before setting the final collection budget. Examine variation by buying-question group and engine. Do not present repeated outputs as independent consumers, and do not apply a simplistic confidence interval that ignores shared prompts, collection days or changes in the platform.

If the observed movement is too small or inconsistent to support the decision, report inconclusive rather than increasing the precision of the dashboard.