Stage 6: How do you tell improvement from sampling noise?

Compare the same questions under comparable conditions and inspect the distribution, not just the average. The design should make it difficult to declare victory by changing the test.

Before comparing waves, confirm:

• Prompt wording and cohort membership are unchanged.
• Surfaces and collection routes are unchanged, or differences are flagged.
• Country, language, account and personalisation settings are comparable.
• Repetition counts and missing-result handling are consistent.
• Brand aliases, competitor set and scoring rules are unchanged.
• Deployment dates and unrelated marketing activity are documented.
• Platform changes and native-reporting anomalies have been checked.

Show per-surface results and prompt-level recurrence. If collection is uneven, present a matched comparison set and separately disclose excluded observations; do not let extra repetitions for favourable prompts dominate the result.

Should you use a control group?

Where practical, leave a comparable group of non-urgent pages or intent clusters unchanged until a later phase. Record why they were selected and check whether changes elsewhere could affect them.

Treat this as a comparison group, not automatic proof of causality. Site-wide changes, external coverage and engine updates can affect both groups. Never delay correction of a serious error for experimental convenience.

For a material budget decision, ask a statistician or experienced measurement analyst to evaluate uncertainty and dependence across prompts, dates and repetitions. The reviewed uncertainty research demonstrates why point estimates can mislead, but its limited topics and observation window do not establish a ready-made confidence calculation for your business. (arxiv.org)

What should success look like?

Agree the decision rule before implementation. A proposed rule could require:

1. Priority technical or factual problems have been demonstrably resolved.
2. Improvement appears in several commercially important intent groups, not one repeatedly sampled question.
3. The direction persists across at least two post-change waves.
4. Accuracy has not deteriorated.
5. The team can explain material collection or platform changes.
6. The next phase has a credible business use and an implementation owner.

These are managerial review criteria—not a statistical significance test or a promised return.