EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes

TL;DR

EvidenceOutcomes is a new, high-quality annotated dataset of clinically meaningful outcomes from biomedical literature, designed to improve extraction and synthesis of evidence in evidence-based medicine.

Contribution

We created a large, annotated corpus with high inter-rater agreement and developed a fine-tuned PubMedBERT model achieving strong performance for extracting outcomes.

Findings

High inter-rater agreement of 0.76 on annotations

PubMedBERT achieved an F1-score of 0.69 at entity level

Dataset serves as a benchmark for future ML algorithms

Abstract

The fundamental process of evidence extraction and synthesis in evidence-based medicine involves extracting PICO (Population, Intervention, Comparison, and Outcome) elements from biomedical literature. However, Outcomes, being the most complex elements, are often neglected or oversimplified in existing benchmarks. To address this issue, we present EvidenceOutcomes, a novel, large, annotated corpus of clinically meaningful outcomes extracted from biomedical literature. We first developed a robust annotation guideline for extracting clinically meaningful outcomes from text through iteration and discussion with clinicians and Natural Language Processing experts. Then, three independent annotators annotated the Results and Conclusions sections of a randomly selected sample of 500 PubMed abstracts and 140 PubMed abstracts from the existing EBM-NLP corpus. This resulted in EvidenceOutcomes with high-quality annotations of an inter-rater agreement of 0.76. Additionally, our fine-tuned PubMedBERT model, applied to these 500 PubMed abstracts, achieved an F1-score of 0.69 at the entity level and 0.76 at the token level on the subset of 140 PubMed abstracts from the EBM-NLP corpus. EvidenceOutcomes can serve as a shared benchmark to develop and test future machine learning algorithms to extract clinically meaningful outcomes from biomedical abstracts.