Statistical NLP for Optimization of Clinical Trial Success Prediction in Pharmaceutical R&D

TL;DR

This paper develops NLP-based models, including a BioBERT-based classifier, to predict clinical trial success in neuroscience, significantly improving prediction accuracy and reducing error compared to traditional methods.

Contribution

It introduces a novel NLP-enabled probabilistic classifier using domain-specific language models for clinical trial success prediction in neuroscience.

Findings

BioBERT model achieved ROC-AUC of 0.74, outperforming traditional models.

NLP features improved prediction accuracy and calibration.

Predictions were 70% better than industry benchmarks.

Abstract

This work presents the development and evaluation of an NLP-enabled probabilistic classifier designed to estimate the probability of technical and regulatory success (pTRS) for clinical trials in the field of neuroscience. While pharmaceutical R&D is plagued by high attrition rates and enormous costs, particularly within neuroscience, where success rates are below 10%, timely identification of promising programs can streamline resource allocation and reduce financial risk. Leveraging data from the ClinicalTrials.gov database and success labels from the recently developed Clinical Trial Outcome dataset, the classifier extracts text-based clinical trial features using statistical NLP techniques. These features were integrated into several non-LLM frameworks (logistic regression, gradient boosting, and random forest) to generate calibrated probability scores. Model performance was assessed on a retrospective dataset of 101,145 completed clinical trials spanning 1976-2024, achieving an overall ROC-AUC of 0.64. An LLM-based predictive model was then built using BioBERT, a domain-specific language representation encoder. The BioBERT-based model achieved an overall ROC-AUC of 0.74 and a Brier Score of 0.185, indicating its predictions had, on average, 40% less squared error than would be observed using industry benchmarks. The BioBERT-based model also made trial outcome predictions that were superior to benchmark values 70% of the time overall. By integrating NLP-driven insights into drug development decision-making, this work aims to enhance strategic planning and optimize investment allocation in neuroscience programs.