Scorecard · No. 08 / 17
What should count as a mention, citation or recommendation?
Last updated
Contract for a metric dictionary before collecting the baseline. The following are proposed operational definitions for this scorecard, not a claim that vendors already use identical definitions.
| Measure | Proposed scoring rule | What it must not imply |
|---|---|---|
| Brand mention frequency | Valid answers containing an unambiguous reference to the brand ÷ valid answers in the specified segment | Positive endorsement or a purchase recommendation |
| Recommendation frequency | Valid answers explicitly presenting the brand as suitable for the stated buying need ÷ valid answers to preclassified recommendation prompts | That every mention, comparison-table entry or negative example is a recommendation |
| Owned-domain citation frequency | Valid answers linking at least once to an approved owned domain ÷ valid answers | That the brand was recommended or that anyone clicked |
| Third-party source association | Separately record whether a linked third-party page contains a verified brand reference and its context | That the page caused the answer or endorses the brand |
| Prompt recommendation coverage | Prompts receiving at least one recommendation ÷ prompts completing the prespecified run schedule | Consistency across runs; show recommendation frequency alongside it |
| Competitive recommendation share | Brand recommendation events ÷ recommendation events for the fixed, disclosed comparison set | Total US market share or the share of all AI users |
| Material claim error rate | Material checked claims contradicted by approved dated evidence ÷ material claims checked | That unchecked claims are accurate; report unverified claims separately |
For competitive share, state whether the denominator includes only named competitors or an additional “other suppliers” category. Keep that definition fixed. Report each brand’s answer-level frequency as well: several brands can be recommended in the same answer.
For Google AI Overviews, record three different states:
- A valid search result with an AI Overview.
- A valid search result without an AI Overview.
- A failed or incomplete collection.
Report both the AI Overview trigger rate and visibility conditional on an Overview appearing. For an overall tested-search measure, a valid no-Overview result contributes no AI Overview recommendation; a collection failure remains unknown. Google’s documentation confirms that AI Overviews do not trigger for every query. (developers.google.com)
Never silently convert missing observations into brand absence. Publish completion rates by surface and period. If one period has materially different coverage, provide a matched-panel comparison rather than quietly changing the mix of prompts behind the headline.
Sources cited on this page
- developers.google.com developers.google.com/search/docs/appearance/ai-features