Revisiting Active Sequential Prediction-Powered Mean Estimation

TL;DR

This paper investigates active sequential mean estimation, proposing a new analysis and showing how query probabilities converge under certain learning strategies, supported by simulations.

Contribution

It provides a non-asymptotic, data-dependent confidence bound analysis and insights into query probability convergence in active learning.

Findings

The smallest confidence width occurs when the constant probability dominates.

The confidence interval bound is data-dependent and non-asymptotic.

Query probability converges to the maximum constraint when using no-regret learning.

Abstract

In this work, we revisit the problem of active sequential prediction-powered mean estimation, where at each round one must decide the query probability of the ground-truth label upon observing the covariates of a sample. Furthermore, if the label is not queried, the prediction from a machine learning model is used instead. Prior work proposed an elegant scheme that determines the query probability by combining an uncertainty-based suggestion with a constant probability that encodes a soft constraint on the query probability. We explored different values of the mixing parameter and observed an intriguing empirical pattern: the smallest confidence width tends to occur when the weight on the constant probability is close to one, thereby reducing the influence of the uncertainty-based component. Motivated by this observation, we develop a non-asymptotic analysis of the estimator and establish a data-dependent bound on its confidence interval. Our analysis further suggests that when a no-regret learning approach is used to determine the query probability and control this bound, the query probability converges to the constraint of the max value of the query probability when it is chosen obliviously to the current covariates. We also conduct simulations that corroborate these theoretical findings.