Stability and Accuracy Trade-offs in Statistical Estimation

TL;DR

This paper investigates the fundamental trade-offs between stability and accuracy in statistical estimation, establishing bounds and optimal estimators to understand how stability constraints impact estimation performance.

Contribution

It provides the first comprehensive analysis of stability-accuracy trade-offs using a decision-theoretic approach, introducing optimal estimators under stability constraints.

Findings

Average-case stability is a weaker restriction than worst-case stability.

Lower bounds on estimation accuracy under stability constraints are established.

Optimal stable estimators are developed for key estimation problems.

Abstract

Algorithmic stability is a central concept in statistics and learning theory that measures how sensitive an algorithm's output is to small changes in the training data. Stability plays a crucial role in understanding generalization, robustness, and replicability, and a variety of stability notions have been proposed in different learning settings. However, while stability entails desirable properties, it is typically not sufficient on its own for statistical learning -- and indeed, it may be at odds with accuracy, since an algorithm that always outputs a constant function is perfectly stable but statistically meaningless. Thus, it is essential to understand the potential statistical cost of stability. In this work, we address this question by adopting a statistical decision-theoretic perspective, treating stability as a constraint in estimation. Focusing on two representative notions-worst-case stability and average-case stability-we first establish general lower bounds on the achievable estimation accuracy under each type of stability constraint. We then develop optimal stable estimators for four canonical estimation problems, including several mean estimation and regression settings. Together, these results characterize the optimal trade-offs between stability and accuracy across these tasks. Our findings formalize the intuition that average-case stability imposes a qualitatively weaker restriction than worst-case stability, and they further reveal that the gap between these two can vary substantially across different estimation problems.