Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges

TL;DR

This paper introduces TrialsSummarizer, a system that automatically retrieves and summarizes relevant clinical trial evidence for specific queries, highlighting current challenges in ensuring summary accuracy and transparency.

Contribution

The paper presents a prototype system combining retrieval, ranking, and neural summarization with architectures aimed at improving transparency in clinical evidence summarization.

Findings

Summaries are fluent and relevant but often contain unsupported statements.

Multi-headed architecture offers potential for better transparency.

Current models are not yet suitable for clinical use due to accuracy issues.

Abstract

We present TrialsSummarizer, a system that aims to automatically summarize evidence presented in the set of randomized controlled trials most relevant to a given query. Building on prior work, the system retrieves trial publications matching a query specifying a combination of condition, intervention(s), and outcome(s), and ranks these according to sample size and estimated study quality. The top-k such studies are passed through a neural multi-document summarization system, yielding a synopsis of these trials. We consider two architectures: A standard sequence-to-sequence model based on BART and a multi-headed architecture intended to provide greater transparency to end-users. Both models produce fluent and relevant summaries of evidence retrieved for queries, but their tendency to introduce unsupported statements render them inappropriate for use in this domain at present. The proposed architecture may help users verify outputs allowing users to trace generated tokens back to inputs.