Contra-Analysis: Prioritizing Meaningful Effect Size in Scientific Research

TL;DR

This paper introduces contra-analysis, a method for comparing effect sizes across diverse studies using credible intervals, to better prioritize scientific research based on meaningful effects.

Contribution

It proposes a new approach and visualization tool, the contra plot, for assessing and ranking effect sizes across heterogeneous studies to improve research prioritization.

Findings

Contra plots enable effective ranking of interventions by effect size.

The method facilitates threshold setting for meaningful effects.

Application to biomedical data demonstrates practical utility.

Abstract

At every phase of scientific research, scientists must decide how to allocate limited resources to pursue the research inquiries with the greatest potential. This prioritization dictates which controlled interventions are studied, awarded funding, published, reproduced with repeated experiments, investigated in related contexts, and translated for societal use. There are many factors that influence this decision-making, but interventions with larger effect size are often favored because they exert the greatest influence on the system studied. To inform these decisions, scientists must compare effect size across studies with dissimilar experiment designs to identify the interventions with the largest effect. These studies are often only loosely related in nature, using experiments with a combination of different populations, conditions, timepoints, measurement techniques, and experiment models that measure the same phenomenon with a continuous variable. We name this assessment contra-analysis and propose to use credible intervals of the relative difference in means to compare effect size across studies in a meritocracy between competing interventions. We propose a data visualization, the contra plot, that allows scientists to score and rank effect size between studies that measure the same phenomenon, aid in determining an appropriate threshold for meaningful effect, and perform hypothesis tests to determine which interventions have meaningful effect size. We illustrate the use of contra plots with real biomedical research data. Contra-analysis promotes a practical interpretation of effect size and facilitates the prioritization of scientific research.