Improving Zero-Shot Detection of Low Prevalence Chest Pathologies using Domain Pre-trained Language Models

TL;DR

This paper explores how domain pre-trained language models can enhance zero-shot detection of rare chest pathologies in X-ray images, showing improved performance on low-prevalence diseases despite some degradation on common ones.

Contribution

It demonstrates that replacing BERT weights with domain-specific pre-trained models improves zero-shot detection of rare diseases in chest X-rays, suggesting ensemble approaches for optimal results.

Findings

Pre-trained domain-specific models outperform generic models on low-prevalence pathologies.

Replacing BERT weights degrades performance on common pathologies.

Ensemble models could leverage strengths of different models for better detection.

Abstract

Recent advances in zero-shot learning have enabled the use of paired image-text data to replace structured labels, replacing the need for expert annotated datasets. Models such as CLIP-based CheXzero utilize these advancements in the domain of chest X-ray interpretation. We hypothesize that domain pre-trained models such as CXR-BERT, BlueBERT, and ClinicalBERT offer the potential to improve the performance of CLIP-like models with specific domain knowledge by replacing BERT weights at the cost of breaking the original model's alignment. We evaluate the performance of zero-shot classification models with domain-specific pre-training for detecting low-prevalence pathologies. Even though replacing the weights of the original CLIP-BERT degrades model performance on commonly found pathologies, we show that pre-trained text towers perform exceptionally better on low-prevalence diseases. This motivates future ensemble models with a combination of differently trained language models for maximal performance.