Entropy-Regularized Inference: A Predictive Approach

TL;DR

This paper introduces an entropy-regularized framework for predictive inference, balancing accuracy and informational complexity, and characterizes optimal rules using information-theoretic measures.

Contribution

It uniquely derives the logarithmic score and Shannon mutual information from axioms, linking entropy regularization to rational inattention and weak identification diagnostics.

Findings

Entropy regularization stabilizes predictions in weakly identified models.

Optimal predictive rules follow a Gibbs form under the proposed axioms.

The framework connects classical diagnostics with entropy-regularized inference.

Abstract

Predictive inference requires balancing statistical accuracy against informational complexity, yet the choice of complexity measure is usually imposed rather than derived. We treat econometric objects as predictive rules, mappings from information to reported predictive distributions, and impose three structural requirements on evaluation: locality, strict propriety, and coherence under aggregation (coarsening/refinement) of outcome categories. These axioms characterize (uniquely, up to affine transformations) the logarithmic score and induce Shannon mutual information (Kullback-Leibler divergence) as the corresponding measure of predictive complexity. The resulting entropy-regularized prediction problem admits Gibbs-form optimal rules, and we establish an essentially complete-class result for the admissible rules we study under joint risk-complexity dominance. Rational inattention emerges as the constrained dual, corresponding to frontier points with binding information capacity. The entropy penalty contributes additive curvature to the predictive criterion; in weakly identified settings, such as weak instruments in IV regression, where the unregularized objective is flat, this curvature stabilizes the predictive criterion. We derive a local quadratic (LAQ) expansion connecting entropy regularization to classical weak-identification diagnostics.