The typicality principle and its implications for statistics and data science

TL;DR

This paper introduces the typicality principle in data science, emphasizing that data's typicality relative to a theory is crucial for valid inference, and proposes a new regularization method based on goodness-of-fit testing that outperforms traditional approaches.

Contribution

It formulates the typicality principle as a new conceptual framework and develops a regularization strategy for parameter estimation grounded in goodness-of-fit testing.

Findings

The regularization strategy performs well in challenging estimation scenarios.

Typicality-based methods improve over maximum likelihood estimation.

The principle enhances uncertainty quantification in data analysis.

Abstract

A central focus of data science is the transformation of empirical evidence into knowledge. As such, the key insights and scientific attitudes of deep thinkers like Fisher, Popper, and Tukey are expected to inspire exciting new advances in machine learning and artificial intelligence in years to come. Along these lines, the present paper advances a novel {\em typicality principle} which states, roughly, that if the observed data is sufficiently ``atypical'' in a certain sense relative to a posited theory, then that theory is unwarranted. This emphasis on typicality brings familiar but often overlooked background notions like model-checking to the inferential foreground. One instantiation of the typicality principle is in the context of parameter estimation, where we propose a new typicality-based regularization strategy that leans heavily on goodness-of-fit testing. The effectiveness of this new regularization strategy is illustrated in three non-trivial examples where ordinary maximum likelihood estimation fails miserably. We also demonstrate how the typicality principle fits within a bigger picture of reliable and efficient uncertainty quantification.