Specialised or Generic? Tokenization Choices for Radiology Language Models

TL;DR

This study systematically compares different tokenizers for radiology language models, showing that domain-specific vocabularies improve summarization performance, reduce memory use, and are beneficial especially when models are trained from scratch.

Contribution

It provides the first comprehensive comparison of general, medical, and domain-specific tokenizers in radiology, highlighting the advantages of domain-specific vocabularies.

Findings

Domain-specific tokenizers outperform general ones in radiology report summarization.

Pre-training reduces performance gaps between tokenizers.

Domain-specific tokenizers lower memory requirements and sequence lengths.

Abstract

The vocabulary used by language models (LM) - defined by the tokenizer - plays a key role in text generation quality. However, its impact remains under-explored in radiology. In this work, we address this gap by systematically comparing general, medical, and domain-specific tokenizers on the task of radiology report summarisation across three imaging modalities. We also investigate scenarios with and without LM pre-training on PubMed abstracts. Our findings demonstrate that medical and domain-specific vocabularies outperformed widely used natural language alternatives when models are trained from scratch. Pre-training partially mitigates performance differences between tokenizers, whilst the domain-specific tokenizers achieve the most favourable results. Domain-specific tokenizers also reduce memory requirements due to smaller vocabularies and shorter sequences. These results demonstrate that adapting the vocabulary of LMs to the clinical domain provides practical benefits, including improved performance and reduced computational demands, making such models more accessible and effective for both research and real-world healthcare settings.