FMRI Clustering in AFNI: False Positive Rates Redux

TL;DR

This paper revisits false positive rates in FMRI cluster analysis, clarifies misconceptions from prior reports, and presents updates to AFNI that improve FPR control, emphasizing the importance of study design and analysis choices.

Contribution

It critically evaluates previous claims of inflated FPRs, demonstrates that AFNI's issues were overstated, and introduces new methods to better control false positives in FMRI analysis.

Findings

Parametric methods show some FPR inflation but not as severe as reported.

AFNI's FPR issues were overstated; updates have improved control.

Permutation methods yield FPRs close to nominal levels.

Abstract

Recent reports of inflated false positive rates (FPRs) in FMRI group analysis tools by Eklund et al. (2016) have become a large topic within (and outside) neuroimaging. They concluded that: existing parametric methods for determining statistically significant clusters had greatly inflated FPRs ("up to 70%," mainly due to the faulty assumption that the noise spatial autocorrelation function is Gaussian- shaped and stationary), calling into question potentially "countless" previous results; in contrast, nonparametric methods, such as their approach, accurately reflected nominal 5% FPRs. They also stated that AFNI showed "particularly high" FPRs compared to other software, largely due to a bug in 3dClustSim. We comment on these points using their own results and figures and by repeating some of their simulations. Briefly, while parametric methods show some FPR inflation in those tests (and assumptions of Gaussian-shaped spatial smoothness also appear to be generally incorrect), their emphasis on reporting the single worst result from thousands of simulation cases greatly exaggerated the scale of the problem. Importantly, FPR statistics depend on "task" paradigm and voxelwise p-value threshold; as such, we show how results of their study provide useful suggestions for FMRI study design and analysis, rather than simply a catastrophic downgrading of the field's earlier results. Regarding AFNI (which we maintain), 3dClustSim's bug-effect was greatly overstated - their own results show that AFNI results were not "particularly" worse than others. We describe further updates in AFNI for characterizing spatial smoothness more appropriately (greatly reducing FPRs, though some remain >5%); additionally, we outline two newly implemented permutation/randomization-based approaches producing FPRs clustered much more tightly about 5% for voxelwise p<=0.01.