Advancing Pancreatic Cancer Prediction with a Next Visit Token Prediction Head on top of Med-BERT

TL;DR

This paper improves pancreatic cancer prediction by reformulating the task to align with Med-BERT's pretraining, significantly boosting accuracy especially in small sample scenarios.

Contribution

It introduces a novel task reformulation approach that aligns disease prediction with Med-BERT's pretraining objectives, enhancing performance in few-shot and larger datasets.

Findings

Reformulating as a token prediction task slightly improves accuracy.

Next visit mask token prediction outperforms binary classification by 3-7%.

Alignment with pretraining objectives enhances disease prediction capabilities.

Abstract

Background: Recently, numerous foundation models pretrained on extensive data have demonstrated efficacy in disease prediction using Electronic Health Records (EHRs). However, there remains some unanswered questions on how to best utilize such models especially with very small fine-tuning cohorts. Methods: We utilized Med-BERT, an EHR-specific foundation model, and reformulated the disease binary prediction task into a token prediction task and a next visit mask token prediction task to align with Med-BERT's pretraining task format in order to improve the accuracy of pancreatic cancer (PaCa) prediction in both few-shot and fully supervised settings. Results: The reformulation of the task into a token prediction task, referred to as Med-BERT-Sum, demonstrates slightly superior performance in both few-shot scenarios and larger data samples. Furthermore, reformulating the prediction task as a Next Visit Mask Token Prediction task (Med-BERT-Mask) significantly outperforms the conventional Binary Classification (BC) prediction task (Med-BERT-BC) by 3% to 7% in few-shot scenarios with data sizes ranging from 10 to 500 samples. These findings highlight that aligning the downstream task with Med-BERT's pretraining objectives substantially enhances the model's predictive capabilities, thereby improving its effectiveness in predicting both rare and common diseases. Conclusion: Reformatting disease prediction tasks to align with the pretraining of foundation models enhances prediction accuracy, leading to earlier detection and timely intervention. This approach improves treatment effectiveness, survival rates, and overall patient outcomes for PaCa and potentially other cancers.