Meaningful Associations Redux: Quantifying and interpreting effect size in the context of the Adolescent Brain and Cognitive Development study

TL;DR

This paper discusses how to interpret effect sizes in large-scale studies like the ABCD Study to improve scientific rigor and reproducibility.

Contribution

The paper introduces a four-part framework for evaluating effect sizes to enhance interpretation and communication in large-scale developmental research.

Findings

Large datasets like ABCD can produce statistically significant but trivial effects.

A framework is proposed to define meaningful effect sizes and improve scientific practices.

Visualizing results enhances interpretation and communication in developmental research.

Abstract

The Adolescent Brain Cognitive Development (ABCD) Study represents a pioneering initiative that aims to unravel the complexities of behavioral and neural development in youth. In this paper, we address the challenges inherent in extracting meaningful insights from the extensive data compiled by the ABCD initiative. Our focus is on advocating for best practices in reproducible research, interpretation of effect size, and reporting of principled results. Central to this discourse is a detailed examination of effect sizes within the expansive ABCD dataset, and how they can be meaningfully interpreted in the context of large-scale research. We describe the hurdles associated with transitioning from conventional small-sample studies to the opportunities and challenges of large samples, including the phenomenon of statistically significant but practically trivial effects. To promote transparent and rigorous inference, we present a four-part framework to evaluate observed effects: researchers should define a smallest effect size of interest (SESOI), compare estimates to relevant benchmarks, test whether observed effects exceed meaningful thresholds (e.g., through equivalence testing), and visualize results to enhance interpretation and communication. Applying this framework yields a clearer, more cumulative understanding of effect size interpretation and contributes substantively to the refinement of scientific practices within adolescent brain and cognitive development research.

Figures