Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models

TL;DR

This paper evaluates the ability of large language models to automatically extract numerical results from randomized controlled trial reports, aiming to enable fully automatic meta-analyses, and identifies current limitations especially with complex outcomes.

Contribution

It introduces a new annotated dataset and assesses LLM performance on extracting numerical findings, highlighting both potential and current limitations for automatic meta-analysis.

Findings

LLMs perform well on simple binary outcomes

Performance drops on complex, inference-requiring outcome measures

Larger LLMs with longer input capacity are promising for automation

Abstract

Meta-analyses statistically aggregate the findings of different randomized controlled trials (RCTs) to assess treatment effectiveness. Because this yields robust estimates of treatment effectiveness, results from meta-analyses are considered the strongest form of evidence. However, rigorous evidence syntheses are time-consuming and labor-intensive, requiring manual extraction of data from individual trials to be synthesized. Ideally, language technologies would permit fully automatic meta-analysis, on demand. This requires accurately extracting numerical results from individual trials, which has been beyond the capabilities of natural language processing (NLP) models to date. In this work, we evaluate whether modern large language models (LLMs) can reliably perform this task. We annotate (and release) a modest but granular evaluation dataset of clinical trial reports with numerical findings attached to interventions, comparators, and outcomes. Using this dataset, we evaluate the performance of seven LLMs applied zero-shot for the task of conditionally extracting numerical findings from trial reports. We find that massive LLMs that can accommodate lengthy inputs are tantalizingly close to realizing fully automatic meta-analysis, especially for dichotomous (binary) outcomes (e.g., mortality). However, LLMs -- including ones trained on biomedical texts -- perform poorly when the outcome measures are complex and tallying the results requires inference. This work charts a path toward fully automatic meta-analysis of RCTs via LLMs, while also highlighting the limitations of existing models for this aim.