Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis

TL;DR

This paper introduces a hybrid tuning strategy to adapt vision-language models for ultrasound analysis, effectively handling ultrasound-specific noise and artifacts, and demonstrating superior performance across multiple datasets.

Contribution

A novel hybrid tuning method that preserves pre-trained model knowledge while incorporating ultrasound-specific modules for improved analysis.

Findings

Significantly outperforms existing adapters and VLFMs in segmentation and classification.

Exhibits high data efficiency in few-shot learning scenarios.

Shows robust generalization across different datasets.

Abstract

Vision-Language Foundation Models (VLFMs) exhibit remarkable generalization, yet their direct application to medical ultrasound is severely hindered by a profound modality gap. The unique acoustic physics of ultrasound, characterized by speckle noise, shadowing, and heterogeneous textures, often degrades the performance of off-the-shelf VLFMs. To bridge this gap, we propose a novel Hybrid Tuning (HT) strategy for the parameter-efficient adaptation of CLIP-based models to ultrasound analysis. Instead of updating the pre-trained weights, HT freezes the visual backbone and integrates a specialized lightweight adapter. This adapter features a Frequency Filtering module to suppress domain-specific periodic artifacts and a Noise Estimation module to dynamically calibrate feature representations. Extensive evaluations across six multi-center datasets demonstrate that our HT-enhanced models significantly outperform existing state-of-the-art adapters and medical VLFMs in both segmentation and classification tasks. Notably, HT exhibits exceptional data efficiency in few-shot scenarios and robust cross-dataset generalization. Our findings prove that preserving pre-trained semantic priors while explicitly modeling ultrasound-specific noise is key to unlocking foundational intelligence in automated ultrasound diagnosis. The source code is available at https://github.com/jinggqu/NextGen-UIA.