10. What would convince you that an agency caused the reported visibility improvement?

A credible case study should explain alternative explanations, not simply show before-and-after screenshots.

Put to: Mike King, Rand Fishkin and an independent statistician.

Why it matters to buyers: NIST’s experimental-design guidance distinguishes designs intended to support comparative conclusions. Seer’s GEO RFP guide also asks buyers to examine what happened after an apparent win, rather than accept its initial presentation. (itl.nist.gov)

Ask: “What evidence would make you comfortable attributing an improvement to the intervention, and what evidence would force you to downgrade that claim?”

A good answer should cover:

• A predeclared outcome and unchanged scoring rules.
• Comparable collection surfaces, prompts and observation windows.
• A dated implementation log and enough pre-intervention observations to examine baseline behaviour.
• A comparison design where feasible, including its limitations and potential spillovers.
• Other changes: product launches, pricing, PR, competitor activity, model changes and seasonality.
• Independent access to the underlying evidence and unsuccessful tests.

Press for: the strongest rival explanation. Ask the expert to describe the result separately as an observation, an association and a causal claim—and identify which level the evidence actually supports.

If a defensible comparison is unavailable, the appropriate conclusion may be “improvement observed after the work”, not “improvement caused by the work”.

Expert quote — pending: ____________________