ROBoto2: An Interactive System and Dataset for LLM-assisted Clinical Trial Risk of Bias Assessment

TL;DR

ROBOT2 is an open-source platform that uses large language models to assist in assessing the risk of bias in clinical trials, streamlining the process with an interactive interface and a new dataset.

Contribution

This paper introduces ROBOTO2, a novel interactive system for LLM-assisted risk of bias assessment and releases a new annotated dataset for benchmarking.

Findings

ROBOT2 effectively streamlines bias assessment workflow.

Benchmark results show varying performance of LLMs on risk of bias tasks.

The dataset enables future research in automated systematic review processes.

Abstract

We present ROBOTO2, an open-source, web-based platform for large language model (LLM)-assisted risk of bias (ROB) assessment of clinical trials. ROBOTO2 streamlines the traditionally labor-intensive ROB v2 (ROB2) annotation process via an interactive interface that combines PDF parsing, retrieval-augmented LLM prompting, and human-in-the-loop review. Users can upload clinical trial reports, receive preliminary answers and supporting evidence for ROB2 signaling questions, and provide real-time feedback or corrections to system suggestions. ROBOTO2 is publicly available at https://roboto2.vercel.app/, with code and data released to foster reproducibility and adoption. We construct and release a dataset of 521 pediatric clinical trial reports (8954 signaling questions with 1202 evidence passages), annotated using both manually and LLM-assisted methods, serving as a benchmark and enabling future research. Using this dataset, we benchmark ROB2 performance for 4 LLMs and provide an analysis into current model capabilities and ongoing challenges in automating this critical aspect of systematic review.