Entropic criterion for model selection

TL;DR

This paper advocates for using relative entropy as a unique, prior-free criterion for model selection, demonstrating its applicability through a physical problem involving simple fluids.

Contribution

It establishes relative entropy as a universal, prior-free model selection criterion based on inductive inference principles, expanding its applicability across fields.

Findings

Relative entropy is shown to be a unique, prior-free criterion.

Application to simple fluids yields promising results.

Supports the use of relative entropy over traditional methods.

Abstract

Model or variable selection is usually achieved through ranking models according to the increasing order of preference. One of methods is applying Kullback-Leibler distance or relative entropy as a selection criterion. Yet that will raise two questions, why uses this criterion and are there any other criteria. Besides, conventional approaches require a reference prior, which is usually difficult to get. Following the logic of inductive inference proposed by Caticha, we show relative entropy to be a unique criterion, which requires no prior information and can be applied to different fields. We examine this criterion by considering a physical problem, simple fluids, and results are promising.