A Statistical Framework for Measuring Reproducibility and Replicability of High‐Throughput Experiments From Multiple Sources

TL;DR

This paper introduces a statistical model to measure how reproducible and replicable results are across multiple high-throughput experiments.

Contribution

A novel statistical framework using a nested copula mixture model to assess reproducibility and replicability across multi-source experiments.

Findings

The model effectively identifies sources of irreproducibility in high-throughput data.

Simulation and real datasets show improved reliability of scientific discoveries using the framework.

Abstract

Replication is essential to reliable and consistent scientific discovery in high‐throughput experiments. Quantifying the replicability of scientific discoveries and identifying sources of irreproducibility have become important tasks for quality control and data integration. In this work we introduce a novel statistical model to measure the reproducibility and replicability of findings from replicate experiments in multi‐source studies. Using a nested copula mixture model that characterizes the interdependence between replication experiments both across and within sources, our method quantifies reproducibility and replicability of each candidate simultaneously in a coherent framework. Through simulation studies, an ENCODE ChIP‐seq dataset and a SEQC RNA‐seq dataset, we demonstrate the effectiveness of our method in diagnosing the source of discordance and improving the reliability of scientific discoveries.

Figures