Comparisons · No. 10 / 13
What should a defensible 90-day evaluation look like?
Last updated
Use the pilot to make a purchase or implementation decision, not to manufacture a success story. The design below is our proposed starting point and should be adapted after the initial repeatability test.
| Stage | Proposed work | Decision gate |
|---|---|---|
| Days 1–15: scope and baseline | Confirm buying questions, approved product facts, surfaces, access methods, native reports and initial repeatability | Is there a recurring, commercially relevant problem? |
| Days 16–30: diagnosis | Inspect cited pages, owned information, technical access and accuracy exceptions | What is observed, what remains a hypothesis, and who can act? |
| Days 31–60: bounded implementation | Deliver a small, logged set of approved changes | Were the changes actually deployed and verified? |
| Days 61–90: follow-up and review | Repeat comparable collection; inspect variation and commercial evidence | Continue, change scope, extend observation or stop? |
Use the 60-prompt core panel separately from exploratory questions. For selected important intents, test alternative wording as a sensitivity check; do not quietly replace poorly performing prompts in the historical series.
Keep an unchanged comparison group where feasible. Record concurrent launches, PR, pricing changes and website releases. Treat a simple before-and-after movement as an association, not causal proof.
This caution is supported by the research on response variability: repeated observations can produce materially different citation distributions, so a favourable endpoint alone is insufficient evidence of intervention effectiveness. (arxiv.org)
How should commercial impact be assessed?
Require three separate evidence streams:
- Answer evidence: comparable mentions, citations, recommendations and factual accuracy.
- Behavioural evidence: native exposure measures, identifiable referrals and relevant on-site actions.
- Buyer evidence: documented discovery conversations, customer interviews and sales feedback.
OpenAI documents the utm_source=chatgpt.com referral parameter. Use it where available, but do not interpret a tracked visit as proof that your optimisation work caused the visit or sale. (help.openai.com)
Do not manufacture “AI revenue” by multiplying a prompt-share percentage by pipeline. If the evidence is insufficient, state what was observed and what remains unknown.
Sources cited on this page
- arxiv.org arxiv.org/abs/2603.08924?utm_source=openai
- help.openai.com help.openai.com/en/articles/12627856-publishers-and-developers-faq%23%3A~%3Atext