What should a defensible pilot look like?

Start with a disclosed diagnostic design, then expand only if the decision requires it. Our suggested starting specification is:

• Questions: 24 prompts across category discovery, requirements, comparisons and shortlisting; distinguish customer-derived wording from analyst proposals.
• Collection: Two relevant surfaces, three fresh-session observations per prompt in each of two measurement windows: 288 planned observations.
• Controls: Record access route, dates, US location settings, language, account tier and observable model information. Use permitted collection methods.
• Scoring: Separate mentions, owned-domain citations, explicit recommendations and material inaccuracies. Review ambiguous cases manually.
• Comparison: Retain unchanged comparison prompts or pages where feasible, and log other marketing and website changes.
• Limit: This is a starting diagnostic, not a statistically powered universal design. Repeated outputs are not independent customers, and a comparison group does not automatically establish causation.

Before commissioning collection, ask the analyst to explain which result would be strong enough to change the proposed decision.