Improving VTE Identification through Language Models from Radiology Reports: A Comparative Study of Mamba, Phi-3 Mini, and BERT

TL;DR

This study compares language models for VTE detection from radiology reports, highlighting Mamba's superior accuracy and efficiency over Phi-3 Mini and BERT, simplifying the process without sacrificing performance.

Contribution

It introduces a Mamba architecture-based classifier that outperforms previous hybrid models and reduces complexity in VTE detection from radiology reports.

Findings

Mamba achieves 97-98% accuracy and F1 score on DVT and PE datasets.

Phi-3 Mini outperforms BERT but is computationally intensive.

Mamba offers a simpler, more efficient solution with comparable or better performance.

Abstract

Venous thromboembolism (VTE) is a critical cardiovascular condition, encompassing deep vein thrombosis (DVT) and pulmonary embolism (PE). Accurate and timely identification of VTE is essential for effective medical care. This study builds upon our previous work, which addressed VTE detection using deep learning methods for DVT and a hybrid approach combining deep learning and rule-based classification for PE. Our earlier approaches, while effective, had two major limitations: they were complex and required expert involvement for feature engineering of the rule set. To overcome these challenges, we utilize the Mamba architecture-based classifier. This model achieves remarkable results, with a 97\% accuracy and F1 score on the DVT dataset and a 98\% accuracy and F1 score on the PE dataset. In contrast to the previous hybrid method on PE identification, the Mamba classifier eliminates the need for hand-engineered rules, significantly reducing model complexity while maintaining comparable performance. Additionally, we evaluated a lightweight Large Language Model (LLM), Phi-3 Mini, in detecting VTE. While this model delivers competitive results, outperforming the baseline BERT models, it proves to be computationally intensive due to its larger parameter set. Our evaluation shows that the Mamba-based model demonstrates superior performance and efficiency in VTE identification, offering an effective solution to the limitations of previous approaches.