Quantifying Ranking Instability Across Evaluation Protocol Axes in Gene Regulatory Network Benchmarking

TL;DR

This paper introduces a diagnostic framework to measure and analyze the stability of method rankings in gene regulatory network benchmarking across different evaluation protocols, revealing insights into the causes of ranking reversals.

Contribution

It provides a systematic method to quantify ranking instability, decomposes reversal causes, and offers practical reporting practices for more stable benchmarking in GRN inference.

Findings

Reversal rates vary across protocol axes, with up to 32.1% for reference network choice.

Observed reversal rates are significantly below random expectations, indicating partial stability.

Reversals are mainly driven by changes in discrimination ability, not base rate inflation.

Abstract

Benchmark rankings are routinely used to justify scientific claims about method quality in gene regulatory network (GRN) inference, yet the stability of these rankings under plausible evaluation protocol choices is rarely examined. We present a systematic diagnostic framework for measuring ranking instability under protocol shift, including decomposition tools that separate base rate effects from discrimination effects. Using existing single cell GRN benchmark outputs across three human tissues and six inference methods, we quantify pairwise reversal rates across four protocol axes: candidate set restriction (16.3 percent, 95 percent CI 11.0 to 23.4 percent), tissue context (19.3 percent), reference network choice (32.1 percent), and symbol mapping policy (0.0 percent). A permutation null confirms that observed reversal rates are far below random order expectations (0.163 versus null mean 0.500), indicating partially stable but non invariant ranking structure. Our decomposition reveals that reversals are driven by changes in the relative discrimination ability of methods rather than by base rate inflation, a finding that challenges a common implicit assumption in GRN benchmarking. We propose concrete reporting practices for stability aware evaluation and provide a diagnostic toolkit for identifying method pairs at risk of reversal.