Diffusion Reconstruction-based Data Likelihood Estimation for Core-Set Selection

TL;DR

This paper introduces a diffusion model-based method for core-set selection that estimates data likelihood through reconstruction deviation, leading to more effective data subset selection for training deep models.

Contribution

The work presents a theoretically grounded diffusion-based likelihood estimation approach for core-set selection, improving over heuristic methods by capturing distributional structures.

Findings

Outperforms existing baseline methods across various selection ratios.

Achieves comparable performance to full-data training using only 50% of the data.

Provides insights into data distribution characteristics through likelihood-informed scoring.

Abstract

Existing core-set selection methods predominantly rely on heuristic scoring signals such as training dynamics or model uncertainty, lacking explicit modeling of data likelihood. This omission may hinder the constructed subset from capturing subtle yet critical distributional structures that underpin effective model training. In this work, we propose a novel, theoretically grounded approach that leverages diffusion models to estimate data likelihood via reconstruction deviation induced by partial reverse denoising. Specifically, we establish a formal connection between reconstruction error and data likelihood, grounded in the Evidence Lower Bound (ELBO) of Markovian diffusion processes, thereby enabling a principled, distribution-aware scoring criterion for data selection. Complementarily, we introduce an efficient information-theoretic method to identify the optimal reconstruction timestep, ensuring that the deviation provides a reliable signal indicative of underlying data likelihood. Extensive experiments on ImageNet demonstrate that reconstruction deviation offers an effective scoring criterion, consistently outperforming existing baselines across selection ratios, and closely matching full-data training using only 50% of the data. Further analysis shows that the likelihood-informed nature of our score reveals informative insights in data selection, shedding light on the interplay between data distributional characteristics and model learning preferences.