# Cross-validation failure: small sample sizes lead to large error bars

## TL;DR

This paper highlights that small sample sizes in neuroimaging studies cause large, often underestimated, error bars in cross-validation, undermining the reliability of predictive models and emphasizing the need for larger samples.

## Contribution

It demonstrates that standard error estimates in cross-validation are often too optimistic and advocates for larger sample sizes to improve reliability in neuroimaging predictive modeling.

## Key findings

- Small sample sizes lead to large error bars (~±10%)
- Standard error across folds underestimates true variability
- Larger samples are necessary for reliable conclusions

## Abstract

Predictive models ground many state-of-the-art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross-validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross-validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg $\pm$10% for 100 samples. The standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarkers or methods developments where, unlike with cognitive neuroimaging MVPA approaches, more samples cannot be acquired by repeating the experiment across many subjects. Solutions to increase sample size must be investigated, tackling possible increases in heterogeneity of the data.