Beyond Point Estimates: Toward Proper Statistical Inferencing and Reporting of Intraclass Correlation Coefficients

TL;DR

This paper emphasizes the importance of proper statistical inference for ICCs in neuroimaging, highlighting common misreporting issues and providing guidelines and tools to improve reliability and validity of ICC-based conclusions.

Contribution

It offers practical guidelines, introduces an online tool for ICC inference, and reassesses recent neuroimaging studies to improve ICC reporting standards.

Findings

Most reviewed studies misreported ICCs or lacked valid statistical inference.

Only 2 out of 11 studies provided valid ICC inferences.

Misuse of ICCs can lead to unreliable scientific conclusions.

Abstract

Reporting test-retest reliability using the intraclass correlation coefficient (ICC) has received increasing attention due to the criticisms of poor transparency and replicability in neuroimaging research, as well as many other biomedical studies. Numerous studies have thus evaluated the reliability of their findings by comparing ICCs, however, they often failed to test statistical differences between ICCs or report confidence intervals. Relying solely on point estimates may preclude valid inference about population-level differences and compromise the reliability of conclusions. To address this issue, this study systematically reviewed the use of ICC in articles published in NeuroImage from 2022 to 2024, highlighting the prevalence of misreporting and misuse of ICCs. We further provide practical guidelines for conducting appropriate statistical inference on ICCs. For practitioners in this area, we introduce an online application for statistical testing and sample size estimation when utilizing ICCs. We recalculated confidence intervals and formally tested ICC values reported in the reviewed articles, thereby reassessing the original inferences. Our results demonstrate that exclusive reliance on point estimates could lead to unreliable or even misleading conclusions. Specifically, only two of the eleven reviewed articles provided unequivocally valid statistical inferences based on ICCs, whereas two articles failed to yield any valid inference at all, raising serious concerns about the replicability of findings in this field. These results underscore the urgent need for rigorous inferential frameworks when reporting and interpreting ICCs.