23. How can we tell whether visibility improved or the supplier is measuring sampling noise?

First audit comparability; only then interpret the change. A larger numerator is not improvement if the denominator, prompts or collection conditions changed.

Illustrative example: four recommendations from 20 answers and eight from 40 answers both equal 20%. Reporting “recommendations doubled” without the denominator is misleading as a visibility-rate claim.

Use this validation sequence:

1. Compare the exact prompt versions and buying-intent mix.
2. Confirm the same surfaces, locations and collection settings.
3. Identify model or access-route changes.
4. Check whether answer failures increased or disappeared.
5. Re-score a sample using unchanged classification rules.
6. Compare equivalent time windows and prompt-level rates.
7. Examine untreated or not-yet-treated themes where feasible.
8. Inspect whether accuracy deteriorated while presence increased.

Research on repeated citation sampling supports treating observed visibility as an estimate with uncertainty. Have an analyst account for clustering by related prompts and collection periods rather than treating every answer as an independent consumer observation. (arxiv.org)

An unchanged comparison group strengthens an argument but does not automatically establish causation: site-wide changes and platform changes can affect both groups. Report conclusions as observed change, plausibly intervention-related change, or stronger causal evidence, according to the actual design.