Consistent information criteria for regularized regression and loss-based learning problems

TL;DR

This paper develops a generalized information criterion framework for consistent model selection in regularized regression and loss-based learning, including uncountable model sets, with theoretical and computational validation.

Contribution

It introduces a unified IC framework for loss-based problems, extending to uncountable model sets, and provides a practical estimation method for GLMs with simulation support.

Findings

The proposed IC framework achieves consistency in model selection.

The method effectively handles uncountable model sets.

Simulation results demonstrate the approach's practical performance.

Abstract

Many problems in statistics and machine learning can be formulated as model selection problems, where the goal is to choose an optimal parsimonious model among a set of candidate models. It is typical to conduct model selection by penalizing the objective function via information criteria (IC), as with the pioneering work by Akaike and Schwarz. Via recent work, we propose a generalized IC framework to consistently estimate general loss-based learning problems. In this work, we propose a consistent estimation method for Generalized Linear Model (GLM) regressions by utilizing the recent IC developments. We advance the generalized IC framework by proposing model selection problems, where the model set consists of a potentially uncountable set of models. In addition to theoretical expositions, our proposal introduces a computational procedure for the implementation of our methods in the finite sample setting, which we demonstrate via an extensive simulation study.