ImmunoFOMO: Are Language Models missing what oncologists see?

TL;DR

This paper evaluates the ability of various language models to identify specific immunotherapy concepts in breast cancer literature, comparing their performance to expert clinicians.

Contribution

It investigates the medical conceptual grounding of language models and finds that pre-trained models can outperform large models in recognizing detailed medical concepts.

Findings

Pre-trained models outperform large language models in identifying specific concepts.

Language models show potential to match expert clinicians in biomedical NLP tasks.

Results highlight the importance of domain-specific training for medical language understanding.

Abstract

Language models (LMs) capabilities have grown with a fast pace over the past decade leading researchers in various disciplines, such as biomedical research, to increasingly explore the utility of LMs in their day-to-day applications. Domain specific language models have already been in use for biomedical natural language processing (NLP) applications. Recently however, the interest has grown towards medical language models and their understanding capabilities. In this paper, we investigate the medical conceptual grounding of various language models against expert clinicians for identification of hallmarks of immunotherapy in breast cancer abstracts. Our results show that pre-trained language models have potential to outperform large language models in identifying very specific (low-level) concepts.