Conflict diagnostics for evidence synthesis in a multiple testing framework

TL;DR

This paper introduces a systematic approach for conflict diagnostics in evidence synthesis models, addressing multiple testing issues to assess consistency across datasets and prior assumptions in Bayesian frameworks.

Contribution

It develops a method for conflict diagnostics that accounts for multiple hypothesis testing in complex probabilistic graphical models used in evidence synthesis.

Findings

Effective conflict diagnostics demonstrated in network meta-analysis of smoking cessation treatments.

Application to HIV prevalence estimation in Poland shows method's practical utility.

Addresses multiple testing problem in model criticism for evidence synthesis.

Abstract

Evidence synthesis models that combine multiple datasets of varying design, to estimate quantities that cannot be directly observed, require the formulation of complex probabilistic models that can be expressed as graphical models. An assessment of whether the different datasets synthesised contribute information that is consistent with each other, and in a Bayesian context, with the prior distribution, is a crucial component of the model criticism process. However, a systematic assessment of conflict suffers from the multiple testing problem, through testing for conflict at multiple locations in a model. We demonstrate the systematic use of conflict diagnostics, while accounting for the multiple hypothesis tests of no conflict at each location in the graphical model. The method is illustrated by a network meta-analysis to estimate treatment effects in smoking cessation programs and an evidence synthesis to estimate HIV prevalence in Poland.