Towards Practical Mean Bounds for Small Samples

TL;DR

This paper introduces a new family of mean bounds for small samples that guarantees coverage under minimal assumptions and often outperforms previous bounds like Anderson's and Hoeffding's in tightness.

Contribution

It presents a novel family of distribution-free mean bounds with guaranteed coverage that are tighter than Anderson's bounds for all samples.

Findings

New bounds outperform Anderson's bounds in simulations

Bounds guarantee coverage for all distributions on an interval

Substantial improvements over previous methods in many cases

Abstract

Historically, to bound the mean for small sample sizes, practitioners have had to choose between using methods with unrealistic assumptions about the unknown distribution (e.g., Gaussianity) and methods like Hoeffding's inequality that use weaker assumptions but produce much looser (wider) intervals. In 1969, Anderson (1969) proposed a mean confidence interval strictly better than or equal to Hoeffding's whose only assumption is that the distribution's support is contained in an interval $[a,b]$. For the first time since then, we present a new family of bounds that compares favorably to Anderson's. We prove that each bound in the family has {\em guaranteed coverage}, i.e., it holds with probability at least $1-\alpha$ for all distributions on an interval $[a,b]$. Furthermore, one of the bounds is tighter than or equal to Anderson's for all samples. In simulations, we show that for many distributions, the gain over Anderson's bound is substantial.