Model Selection and Parameter Inference through Constraints via Sequences of Surrogate Smoothing Functions

TL;DR

This paper introduces a continuous optimization approach for model selection and parameter inference by constructing smooth surrogate functions that replicate discrete criteria like AIC and BIC.

Contribution

It presents a novel method of smoothing discrete model selection objectives, enabling continuous optimization and including a new clustering technique via overparameterization.

Findings

Surrogate functions successfully replicate AIC/BIC optima.

Continuous optimization reduces selection complexity.

Overparameterization-based clustering shows promising results.

Abstract

Models with fewer parameters are often easier to interpret and more robust. Parsimony can be achieved through optimizing objectives like the AIC or BIC, which are functions of the the number of free parameters in the model. Optimizing this discrete objective is a challenge, often relying on discrete optimization. We construct smooth functions with optima that reach the same optima of these objectives but permit continuous rather than discrete optimization, relieving some selection burden. Proofs of convergence are provided and a novel method of clustering through explicit overparamterization shows promising results.