TinyUSFM: Towards Compact and Efficient Ultrasound Foundation Models

TL;DR

TinyUSFM is a lightweight ultrasound foundation model that achieves high performance and efficiency through knowledge distillation, curated data, and domain-specific training strategies, enabling deployment in resource-limited settings.

Contribution

The paper introduces TinyUSFM, the first compact ultrasound foundation model that maintains high versatility and task adaptability using novel data selection and distillation techniques.

Findings

TinyUSFM matches large USFM performance with only 6.36% parameters.

It outperforms vanilla models by 9.45% in classification accuracy.

Achieves 84.91% average classification accuracy across diverse datasets.

Abstract

Foundation models for medical imaging demonstrate superior generalization capabilities across diverse anatomical structures and clinical applications. Their outstanding performance relies on substantial computational resources, limiting deployment in resource-constrained clinical environments. This paper presents TinyUSFM, the first lightweight ultrasound foundation model that maintains superior organ versatility and task adaptability of our large-scale Ultrasound Foundation Model (USFM) through knowledge distillation with strategically curated small datasets, delivering significant computational efficiency without sacrificing performance. Considering the limited capacity and representation ability of lightweight models, we propose a feature-gradient driven coreset selection strategy to curate high-quality compact training data, avoiding training degradation from low-quality redundant images. To preserve the essential spatial and frequency domain characteristics during knowledge transfer, we develop domain-separated masked image modeling assisted consistency-driven dynamic distillation. This novel framework adaptively transfers knowledge from large foundation models by leveraging teacher model consistency across different domain masks, specifically tailored for ultrasound interpretation. For evaluation, we establish the UniUS-Bench, the largest publicly available ultrasound benchmark comprising 8 classification and 10 segmentation datasets across 15 organs. Using only 200K images in distillation, TinyUSFM matches USFM's performance with just 6.36% of parameters and 6.40% of GFLOPs. TinyUSFM significantly outperforms the vanilla model by 9.45% in classification and 7.72% in segmentation, surpassing all state-of-the-art lightweight models, and achieving 84.91% average classification accuracy and 85.78% average segmentation Dice score across diverse medical devices and centers.