An active learning framework for multi-group mean estimation

TL;DR

This paper introduces an active learning framework for estimating means across multiple groups with unknown distributions, focusing on fairness and efficiency in adaptive data collection, and proposes a Variance-UCB algorithm with improved theoretical guarantees.

Contribution

The paper develops a novel active learning approach with a Variance-UCB algorithm for multi-group mean estimation, providing the first general theoretical bounds for variance-based adaptive sampling.

Findings

The Variance-UCB algorithm effectively minimizes estimation noise across groups.

Theoretical bounds on regret significantly improve upon previous results.

Framework applies to diverse distributions with reliable variance estimation.

Abstract

We study a fundamental learning problem over multiple groups with unknown data distributions, where an analyst would like to learn the mean of each group. Moreover, we want to ensure that this data is collected in a relatively fair manner such that the noise of the estimate of each group is reasonable. In particular, we focus on settings where data are collected dynamically, which is important in adaptive experimentation for online platforms or adaptive clinical trials for healthcare. In our model, we employ an active learning framework to sequentially collect samples with bandit feedback, observing a sample in each period from the chosen group. After observing a sample, the analyst updates their estimate of the mean and variance of that group and chooses the next group accordingly. The analyst's objective is to dynamically collect samples to minimize the collective noise of the estimators, measured by the norm of the vector of variances of the mean estimators. We propose an algorithm, Variance-UCB, that sequentially selects groups according to an upper confidence bound on the variance estimate. We provide a general theoretical framework for providing efficient bounds on learning from any underlying distribution where the variances can be estimated reasonably. This framework yields upper bounds on regret that improve significantly upon all existing bounds, as well as a collection of new results for different objectives and distributions than those previously studied.