Exploring the Generalization of Cancer Clinical Trial Eligibility Classifiers Across Diseases

TL;DR

This study evaluates how well NLP models trained on cancer trial eligibility criteria generalize to other disease trials, revealing strengths and limitations, and introduces a new dataset for cross-disease clinical trial classification.

Contribution

It provides a comprehensive dataset and analysis of model generalization across multiple disease types, highlighting the potential and challenges of NLP in clinical trial eligibility classification.

Findings

Models trained on cancer data generalize to autoimmune criteria.

Performance drops on cancer-specific criteria like prior malignancy.

Few-shot learning improves cross-disease classification performance.

Abstract

Clinical trials are pivotal in medical research, and NLP can enhance their success, with application in recruitment. This study aims to evaluate the generalizability of eligibility classification across a broad spectrum of clinical trials. Starting with phase 3 cancer trials, annotated with seven eligibility exclusions, then to determine how well models can generalize to non-cancer and non-phase 3 trials. To assess this, we have compiled eligibility criteria data for five types of trials: (1) additional phase 3 cancer trials, (2) phase 1 and 2 cancer trials, (3) heart disease trials, (4) type 2 diabetes trials, and (5) observational trials for any disease, comprising 2,490 annotated eligibility criteria across seven exclusion types. Our results show that models trained on the extensive cancer dataset can effectively handle criteria commonly found in non-cancer trials, such as autoimmune diseases. However, they struggle with criteria disproportionately prevalent in cancer trials, like prior malignancy. We also experiment with few-shot learning, demonstrating that a limited number of disease-specific examples can partially overcome this performance gap. We are releasing this new dataset of annotated eligibility statements to promote the development of cross-disease generalization in clinical trial classification.