An index of effective number of variables for uncertainty and reliability analysis in model selection problems

TL;DR

This paper introduces the ENV index, a new measure for effective variable count in model selection, which improves existing methods and can be used with various criteria, validated through experiments with real data.

Contribution

The paper proposes the ENV index, a novel measure for effective number of variables, addressing limitations of existing methods and enhancing model selection procedures.

Findings

ENV index outperforms traditional elbow methods

Provides confidence measures for model selection

Compatible with AIC, BIC, and other criteria

Abstract

An index of an effective number of variables (ENV) is introduced for model selection in nested models. This is the case, for instance, when we have to decide the order of a polynomial function or the number of bases in a nonlinear regression, choose the number of clusters in a clustering problem, or the number of features in a variable selection application (to name few examples). It is inspired by the idea of the maximum area under the curve (AUC). The interpretation of the ENV index is identical to the effective sample size (ESS) indices concerning a set of samples. The ENV index improves {drawbacks of} the elbow detectors described in the literature and introduces different confidence measures of the proposed solution. These novel measures can be also employed jointly with the use of different information criteria, such as the well-known AIC and BIC, or any other model selection procedures. Comparisons with classical and recent schemes are provided in different experiments involving real datasets. Related Matlab code is given.