Reconciling Model Selection and Prediction

TL;DR

This paper investigates the trade-off between model selection consistency and prediction accuracy, showing that the apparent dichotomy is due to extreme cases and can be reconciled under realistic conditions.

Contribution

It characterizes the extreme parameter set causing the dichotomy and demonstrates that excluding these cases allows for both consistency and minimax optimal prediction.

Findings

The dichotomy is due to a pathological parameter set.

Excluding extreme parameters enables simultaneous consistency and minimaxity.

The set causing the dichotomy is shown to be negligible in practical scenarios.

Abstract

It is known that there is a dichotomy in the performance of model selectors. Those that are consistent (having the "oracle property") do not achieve the asymptotic minimax rate for prediction error. We look at this phenomenon closely, and argue that the set of parameters on which this dichotomy occurs is extreme, even pathological, and should not be considered when evaluating model selectors. We characterize this set, and show that, when such parameters are dismissed from consideration, consistency and asymptotic minimaxity can be attained simultaneously.