Model Selection with the Loss Rank Principle

TL;DR

The paper introduces the Loss Rank Principle (LoRP), a novel model selection method for regression and classification that chooses models based on how many hypothetical datasets they fit better, without relying on stochastic noise models.

Contribution

It proposes LoRP, a new model selection criterion that is applicable to any non-parametric regressor and depends solely on the regression functions and loss function.

Findings

LoRP effectively selects appropriate model complexity.

LoRP is applicable without a stochastic noise model.

LoRP outperforms traditional criteria like AIC, BIC, MDL.

Abstract

A key issue in statistics and machine learning is to automatically select the "right" model complexity, e.g., the number of neighbors to be averaged over in k nearest neighbor (kNN) regression or the polynomial degree in regression with polynomials. We suggest a novel principle - the Loss Rank Principle (LoRP) - for model selection in regression and classification. It is based on the loss rank, which counts how many other (fictitious) data would be fitted better. LoRP selects the model that has minimal loss rank. Unlike most penalized maximum likelihood variants (AIC, BIC, MDL), LoRP depends only on the regression functions and the loss function. It works without a stochastic noise model, and is directly applicable to any non-parametric regressor, like kNN.