RoBIn: A Transformer-Based Model For Risk Of Bias Inference With Machine Reading Comprehension

TL;DR

This paper introduces RoBIn, a Transformer-based model that automates the assessment of risk of bias in clinical trial publications by extracting evidence and classifying bias levels, outperforming existing methods.

Contribution

The study presents a novel dataset and two Transformer-based approaches for automated risk of bias inference, advancing machine reading comprehension in healthcare research.

Findings

RoBIn achieves an ROC AUC of 0.83 in bias classification.

Both RoBInGen and RoBInExt outperform traditional and LLM-based methods.

The model effectively distinguishes low from high/unclear risk of bias.

Abstract

Objective: Scientific publications play a crucial role in uncovering insights, testing novel drugs, and shaping healthcare policies. Accessing the quality of publications requires evaluating their Risk of Bias (RoB), a process typically conducted by human reviewers. In this study, we introduce a new dataset for machine reading comprehension and RoB assessment and present RoBIn (Risk of Bias Inference), an innovative model crafted to automate such evaluation. The model employs a dual-task approach, extracting evidence from a given context and assessing the RoB based on the gathered evidence. Methods: We use data from the Cochrane Database of Systematic Reviews (CDSR) as ground truth to label open-access clinical trial publications from PubMed. This process enabled us to develop training and test datasets specifically for machine reading comprehension and RoB inference. Additionally, we created extractive (RoBInExt) and generative (RoBInGen) Transformer-based approaches to extract relevant evidence and classify the RoB effectively. Results: RoBIn is evaluated across various settings and benchmarked against state-of-the-art methods for RoB inference, including large language models in multiple scenarios. In most cases, the best-performing RoBIn variant surpasses traditional machine learning and LLM-based approaches, achieving an ROC AUC of 0.83. Conclusion: Based on the evidence extracted from clinical trial reports, RoBIn performs a binary classification to decide whether the trial is at a low RoB or a high/unclear RoB. We found that both RoBInGen and RoBInExt are robust and have the best results in many settings.