Progressive trajectory matching for medical dataset distillation

TL;DR

This paper introduces a progressive trajectory matching approach for medical dataset distillation, enhancing stability and diversity of synthetic datasets to improve model training while preserving privacy.

Contribution

It proposes a novel progressive trajectory matching strategy and a dynamic overlap mitigation module specifically designed for medical image dataset distillation, along with a new benchmark for evaluation.

Findings

Achieves 8.33% improvement over previous methods on average.

Improves performance by 11.7% when image per class is 2.

Enhances training stability and synthetic dataset diversity.

Abstract

It is essential but challenging to share medical image datasets due to privacy issues, which prohibit building foundation models and knowledge transfer. In this paper, we propose a novel dataset distillation method to condense the original medical image datasets into a synthetic one that preserves useful information for building an analysis model without accessing the original datasets. Existing methods tackle only natural images by randomly matching parts of the training trajectories of the model parameters trained by the whole real datasets. However, through extensive experiments on medical image datasets, the training process is extremely unstable and achieves inferior distillation results. To solve these barriers, we propose to design a novel progressive trajectory matching strategy to improve the training stability for medical image dataset distillation. Additionally, it is observed that improved stability prevents the synthetic dataset diversity and final performance improvements. Therefore, we propose a dynamic overlap mitigation module that improves the synthetic dataset diversity by dynamically eliminating the overlap across different images and retraining parts of the synthetic images for better convergence. Finally, we propose a new medical image dataset distillation benchmark of various modalities and configurations to promote fair evaluations. It is validated that our proposed method achieves 8.33% improvement over previous state-of-the-art methods on average, and 11.7% improvement when ipc=2 (i.e., image per class is 2). Codes and benchmarks will be released.