Entropy-based Characterization of Modeling Constraints

TL;DR

This paper introduces a novel entropy-based framework that derives models directly from phenomenological constraints, enabling data-driven hypothesis testing and model selection without relying on prior parametric assumptions.

Contribution

It presents a perpendicular formulation to traditional MaxEnt approaches, deriving distributions from constraints and unifying model selection and scoring procedures within a data-driven context.

Findings

MaxEnt distribution is a central, typical solution among feasible distributions.

The framework supports hypothesis testing and model selection based solely on data.

Major scoring and selection procedures are recovered and assessed within this entropy-based approach.

Abstract

In most data-scientific approaches, the principle of Maximum Entropy (MaxEnt) is used to a posteriori justify some parametric model which has been already chosen based on experience, prior knowledge or computational simplicity. In a perpendicular formulation to conventional model building, we start from the linear system of phenomenological constraints and asymptotically derive the distribution over all viable distributions that satisfy the provided set of constraints. The MaxEnt distribution plays a special role, as it is the most typical among all phenomenologically viable distributions representing a good expansion point for large-N techniques. This enables us to consistently formulate hypothesis testing in a fully-data driven manner. The appropriate parametric model which is supported by the data can be always deduced at the end of model selection. In the MaxEnt framework, we recover major scores and selection procedures used in multiple applications and assess their ability to capture associations in the data-generating process and identify the most generalizable model. This data-driven counterpart of standard model selection demonstrates the unifying prospective of the deductive logic advocated by MaxEnt principle, while potentially shedding new insights to the inverse problem.