EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data

TL;DR

EchoDFKD introduces a data-free knowledge distillation approach for cardiac ultrasound segmentation, achieving state-of-the-art results using synthetic data and a novel evaluation method that bypasses human annotation.

Contribution

The paper presents a novel data-free knowledge distillation method for ultrasound segmentation that performs comparably to real-data training and introduces an annotation-free evaluation technique.

Findings

Achieves SOTA performance on key cardiac frame identification tasks.

Training on synthetic data yields near real-data performance.

Outperforms five existing methods in most evaluation scenarios.

Abstract

The application of machine learning to medical ultrasound videos of the heart, i.e., echocardiography, has recently gained traction with the availability of large public datasets. Traditional supervised tasks, such as ejection fraction regression, are now making way for approaches focusing more on the latent structure of data distributions, as well as generative methods. We propose a model trained exclusively by knowledge distillation, either on real or synthetical data, involving retrieving masks suggested by a teacher model. We achieve state-of-the-art (SOTA) values on the task of identifying end-diastolic and end-systolic frames. By training the model only on synthetic data, it reaches segmentation capabilities close to the performance when trained on real data with a significantly reduced number of weights. A comparison with the 5 main existing methods shows that our method outperforms the others in most cases. We also present a new evaluation method that does not require human annotation and instead relies on a large auxiliary model. We show that this method produces scores consistent with those obtained from human annotations. Relying on the integrated knowledge from a vast amount of records, this method overcomes certain inherent limitations of human annotator labeling. Code: https://github.com/GregoirePetit/EchoDFKD