Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes

TL;DR

This paper introduces a new coreset selection framework using posterior sampling that improves training speed and generalization in deep learning models by addressing loss landscape challenges.

Contribution

It establishes a connection between posterior sampling and loss landscapes, proposing a smoothed loss function for more stable and effective coreset selection.

Findings

Achieves faster training compared to existing methods

Enhances model generalization across multiple datasets

Provides a novel convergence analysis for sampling-based coreset selection

Abstract

As deep learning models continue to scale, the growing computational demands have amplified the need for effective coreset selection techniques. Coreset selection aims to accelerate training by identifying small, representative subsets of data that approximate the performance of the full dataset. Among various approaches, gradient based methods stand out due to their strong theoretical underpinnings and practical benefits, particularly under limited data budgets. However, these methods face challenges such as naive stochastic gradient descent (SGD) acting as a surprisingly strong baseline and the breakdown of representativeness due to loss curvature mismatches over time. In this work, we propose a novel framework that addresses these limitations. First, we establish a connection between posterior sampling and loss landscapes, enabling robust coreset selection even in high data corruption scenarios. Second, we introduce a smoothed loss function based on posterior sampling onto the model weights, enhancing stability and generalization while maintaining computational efficiency. We also present a novel convergence analysis for our sampling-based coreset selection method. Finally, through extensive experiments, we demonstrate how our approach achieves faster training and enhanced generalization across diverse datasets than the current state of the art.