The use of cross validation in the analysis of designed experiments

TL;DR

This paper empirically evaluates the effectiveness of cross-validation methods, especially LOOCV, in analyzing small, structured designed experiments, comparing them to bootstrap and other model selection techniques.

Contribution

It provides evidence that LOOCV can be useful in small, structured experimental designs, challenging traditional warnings against CV use in such contexts.

Findings

LOOCV often useful in small, structured experiments

k-fold CV performance is inconsistent

LOOCV outperforms bootstrap in some scenarios

Abstract

Cross-validation (CV) is a common method to tune machine learning methods and can be used for model selection in regression as well. Because of the structured nature of small, traditional experimental designs, the literature has warned against using CV in their analysis. The striking increase in the use of machine learning, and thus CV, in the analysis of experimental designs, has led us to empirically study the effectiveness of CV compared to other methods of selecting models in designed experiments, including the little bootstrap. We consider both response surface settings where prediction is of primary interest, as well as screening where factor selection is most important. Overall, we provide evidence that the use of leave-one-out cross-validation (LOOCV) in the analysis of small, structured is often useful. More general $k$-fold CV may also be competitive but its performance is uneven.