Distribution-Free Statistical Dispersion Control for Societal Applications

TL;DR

This paper introduces a new framework for controlling the dispersion of loss distributions in machine learning, ensuring fairness and reliability in societal applications without relying on specific data distributions.

Contribution

It proposes a flexible, distribution-free approach to controlling statistical dispersion measures, extending beyond previous methods focused on expected loss or prediction intervals.

Findings

Effective in toxic comment detection

Applicable to medical imaging

Useful for fair film recommendation

Abstract

Explicit finite-sample statistical guarantees on model performance are an important ingredient in responsible machine learning. Previous work has focused mainly on bounding either the expected loss of a predictor or the probability that an individual prediction will incur a loss value in a specified range. However, for many high-stakes applications, it is crucial to understand and control the dispersion of a loss distribution, or the extent to which different members of a population experience unequal effects of algorithmic decisions. We initiate the study of distribution-free control of statistical dispersion measures with societal implications and propose a simple yet flexible framework that allows us to handle a much richer class of statistical functionals beyond previous work. Our methods are verified through experiments in toxic comment detection, medical imaging, and film recommendation.