2024-12-16

Benchmarking without pretending your sample is bigger than it is

By Nari Go

Hero art for Benchmarking without pretending your sample is bigger than it is

Peer benchmarks seduce leadership decks because they imply certainty. We publish minimum store counts before a benchmark line appears. Below threshold, we print a muted footnote instead of a chart.

Analysts document cohort definitions each quarter because banner mix shifts silently when distributors win or lose routes.

Third, we encourage relative trend lines for thin cohorts rather than absolute rank. A moving average across three waves often tells a truer story than a single snapshot.

We will refuse a benchmark request if merging datasets would de-anonymize a partner. That limitation has cost us short-term revenue and preserved long-term access.

analyticsethicsreporting
Back to field notes