Predictive Hypothesis Identification

TL;DR

This paper introduces a general principle called Predictive Hypothesis Identification (PHI) that unifies various hypothesis evaluation methods in machine learning by focusing on predictive performance, including handling nested hypotheses.

Contribution

The paper proposes PHI as a unifying framework that encompasses existing hypothesis testing and model selection methods, providing a new perspective on predictive inference.

Findings

PHI recovers well-known methods like MAP, ML, MDL, and moment estimation.

PHI can handle nested hypotheses effectively.

PHI offers a unified approach to various hypothesis evaluation techniques.

Abstract

While statistics focusses on hypothesis testing and on estimating (properties of) the true sampling distribution, in machine learning the performance of learning algorithms on future data is the primary issue. In this paper we bridge the gap with a general principle (PHI) that identifies hypotheses with best predictive performance. This includes predictive point and interval estimation, simple and composite hypothesis testing, (mixture) model selection, and others as special cases. For concrete instantiations we will recover well-known methods, variations thereof, and new ones. PHI nicely justifies, reconciles, and blends (a reparametrization invariant variation of) MAP, ML, MDL, and moment estimation. One particular feature of PHI is that it can genuinely deal with nested hypotheses.