Meta-Statistical Learning: Supervised Learning of Statistical Estimators

TL;DR

This paper introduces meta-statistical learning, a supervised neural network-based framework that automates the design of statistical estimators, enabling empirical discovery of estimators for tasks like normality testing and mutual information estimation.

Contribution

It presents a novel amortized learning approach that formulates estimator design as an optimization problem, allowing for empirical, data-driven discovery of statistical estimators.

Findings

Achieved strong results in normality testing and mutual information estimation.

Demonstrated effectiveness with small neural network models.

Provided a new paradigm for automating estimator discovery.

Abstract

Statistical inference, a central tool of science, revolves around the study and the usage of statistical estimators: functions that map finite samples to predictions about unknown distribution parameters. In the frequentist framework, estimators are evaluated based on properties such as bias, variance (for parameter estimation), accuracy, power, and calibration (for hypothesis testing). However, crafting estimators with desirable properties is often analytically challenging, and sometimes impossible, e.g., there exists no universally unbiased estimator for the standard deviation. In this work, we introduce meta-statistical learning, an amortized learning framework that recasts estimator design as an optimization problem via supervised learning. This takes a fully empirical approach to discovering statistical estimators; entire datasets are input to permutation-invariant neural networks, such as Set Transformers, trained to predict the target statistical property. The trained model is the estimator, and can be analyzed through the classical frequentist lens. We demonstrate the approach on two tasks: learning a normality test (classification) and estimating mutual information (regression), achieving strong results even with small models. Looking ahead, this paradigm opens a path to automate the discovery of generalizable and flexible statistical estimators.