2024-12-16
Benchmarking without pretending your sample is bigger than it is
By Nari Go
Peer benchmarks seduce leadership decks because they imply certainty. We publish minimum store counts before a benchmark line appears. Below threshold, we print a muted footnote instead of a chart.
Analysts document cohort definitions each quarter because banner mix shifts silently when distributors win or lose routes.
Third, we encourage relative trend lines for thin cohorts rather than absolute rank. A moving average across three waves often tells a truer story than a single snapshot.
We will refuse a benchmark request if merging datasets would de-anonymize a partner. That limitation has cost us short-term revenue and preserved long-term access.
analyticsethicsreporting
Back to field notes