Cross validation for model selection: a primer with examples from ecology

TL;DR

This paper provides a comprehensive primer on cross-validation techniques for model selection in ecology, emphasizing practical application, technical considerations, and addressing misconceptions, with case studies illustrating its effectiveness.

Contribution

It offers detailed guidance on applying cross-validation in ecological research, including variants, technical nuances, and recommendations, bridging gaps between theory and practice.

Findings

CV is versatile and applicable even without likelihood or parameter counts.

Leave-one-out and approximate CV minimize bias effectively.

Calibrated model selection rules help prevent overfitting.

Abstract

The growing use of model-selection principles in ecology for statistical inference is underpinned by information criteria (IC) and cross-validation (CV) techniques. Although IC techniques, such as Akaike's Information Criterion, have been historically more popular in ecology, CV is a versatile and increasingly used alternative. CV uses data splitting to estimate model scores based on (out-of-sample) predictive performance, which can be used even when it is not possible to derive a likelihood (e.g., machine learning) or count parameters precisely (e.g., mixed-effects models and penalised regression). Here we provide a primer to understanding and applying CV in ecology. We review commonly applied variants of CV, including approximate methods, and make recommendations for their use based on the statistical context. We explain some important -- but often overlooked -- technical aspects of CV, such as bias correction, estimation uncertainty, score selection, and parsimonious selection rules. We also address misconceptions (and truths) about impediments to the use of CV, including computational cost and ease of implementation, and clarify the relationship between CV and information-theoretic approaches to model selection. The paper includes two ecological case studies which illustrate the application of the techniques. We conclude that CV-based model selection should be widely applied to ecological analyses, because of its robust estimation properties and the broad range of situations for which it is applicable. In particular, we recommend using leave-one-out (LOO) or approximate LOO CV to minimise bias, or otherwise K-fold CV using bias correction if K<10. To mitigate overfitting, we recommend calibrated selection via the modified one-standard-error rule which accounts for the predominant cause of overfitting: score-estimation uncertainty.