A Note on Model Selection for Small Sample Regression

TL;DR

This paper analyzes the Direct Eigenvalue Estimator (DEE) for small sample regression, identifies a flaw in its derivation, and proposes an improved class of risk estimators (mDEE) that outperform DEE and ADJ in experiments.

Contribution

It reveals a flaw in DEE's derivation and introduces a new class of risk estimators (mDEE) that enhance performance in small sample regression.

Findings

mDEE often outperforms DEE and ADJ in experiments

DEE's derivation contains an inappropriate part affecting its potential

The new class of estimators improves risk estimation accuracy

Abstract

The risk estimator called "Direct Eigenvalue Estimator" (DEE) is studied. DEE was developed for small sample regression. In contrast to many existing model selection criteria, derivation of DEE requires neither any asymptotic assumption nor any prior knowledge about the noise variance and the noise distribution. It was reported that DEE performed well in small sample cases but DEE performed a little worse than the state-of-the-art ADJ. This seems somewhat counter-intuitive because DEE was developed for specifically regression problem by exploiting available information exhaustively, while ADJ was developed for general setting. In this paper, we point out that the derivation of DEE includes an inappropriate part in spite that the resultant form of DEE is valid in a sense. As its result, DEE cannot derive its potential. We introduce a class of valid risk estimators based on the idea of DEE and show that better risk estimators (mDEE) can be found in the class. By numerical experiments, we verify that mDEE often performs better than or at least equally the original DEE and ADJ.