How do you distinguish visibility lift from sampling noise?

Require evidence that the result survives repeated observations and reasonable changes in test wording. Do not demand identical prose on reruns; demand an auditable collection and scoring process.

The reviewed research identifies both run-to-run variability and sensitivity to prompt formulation. Its limited domains and collection windows mean a buyer should calibrate the method on its own category instead of importing a purported universal sample size. (arxiv.org)

Ask the supplier to complete five checks:

1. Repeatability check: collect the same prompt and surface repeatedly under disclosed conditions.
2. Wording check: test a small, separately reported set of meaning-preserving rephrasings. Keep added requirements separate: changing the requested use case changes the task.
3. Time check: spread observations across more than one collection window.
4. Missingness check: identify whether failures cluster around particular surfaces, prompts or periods.
5. Decision check: show whether the recommended action changes under reasonable alternative aggregation choices.

Require uncertainty reporting appropriate to the design. Do not accept a confidence interval calculated as though every rerun were an independent customer. Ask an analyst to explain the treatment of repeated prompts, shared collection windows and dependence between observations. For a small diagnostic, clearly reported counts and ranges may be more honest than an elaborate significance claim.

NIST recommends documenting biases or statistical variance and checking whether a metric actually measures the intended concept. That is a useful challenge for any supplier promising a precise “AI authority” score. (nvlpubs.nist.gov)

What collection metadata must be exportable?

Request, at minimum:

prompt_id, prompt_version, exact_prompt, intent, surface, access_route, collection_timestamp, requested_location, verified_location_or_limit, language, session_state, model_if_observable, answer_text, citation_urls, collection_status, retry_status, brand_labels, recommendation_labels, scoring_version, reviewer_status

Also request a record of model or interface changes, collection-policy changes and historical reclassification.

For consumer-interface tests, disclose login state, relevant personalisation settings and whether prior conversation context is present. OpenAI documents that saved memories and location information can influence search query formulation; an unspecified “US ChatGPT check” is therefore an incomplete test description. (help.openai.com)