Resolving conflicts between statistical methods by probability combination: Application to empirical Bayes analyses of genomic data

TL;DR

This paper introduces a game-theoretic method for combining statistically plausible distributions without subjective weights, improving robustness in empirical Bayes analyses of genomic data.

Contribution

It proposes a novel distribution combination approach based on a game-theoretic framework that avoids subjective weighting, applicable to diverse statistical methods.

Findings

The method effectively combines conflicting empirical Bayes approaches.

Combined distributions are linear mixes of the most extreme plausible ones.

Weights are balanced to prevent dominance by any single distribution.

Abstract

In the typical analysis of a data set, a single method is selected for statistical reporting even when equally applicable methods yield very different results. Examples of equally applicable methods can correspond to those of different ancillary statistics in frequentist inference and of different prior distributions in Bayesian inference. More broadly, choices are made between parametric and nonparametric methods and between frequentist and Bayesian methods. Rather than choosing a single method, it can be safer, in a game-theoretic sense, to combine those that are equally appropriate in light of the available information. Since methods of combining subjectively assessed probability distributions are not objective enough for that purpose, this paper introduces a method of distribution combination that does not require any assignment of distribution weights. It does so by formalizing a hedging strategy in terms of a game between three players: nature, a statistician combining distributions, and a statistician refusing to combine distributions. The optimal move of the first statistician reduces to the solution of a simpler problem of selecting an estimating distribution that minimizes the Kullback-Leibler loss maximized over the plausible distributions to be combined. The resulting combined distribution is a linear combination of the most extreme of the distributions to be combined that are scientifically plausible. The optimal weights are close enough to each other that no extreme distribution dominates the others. The new methodology is illustrated by combining conflicting empirical Bayes methodologies in the context of gene expression data analysis.