DIET: Learning to Distill Dataset Continually for Recommender Systems

TL;DR

DIET introduces an evolving dataset distillation framework for recommender systems, significantly reducing training data size and computational costs while maintaining performance, thus enabling scalable continual learning.

Contribution

This paper proposes DIET, a novel method for streaming dataset distillation that maintains a dynamic, compact dataset aligned with long-term training dynamics in recommender systems.

Findings

Reduces training data to 1-2% of original size

Speeds up model iteration by up to 60 times

Distilled datasets generalize across different models

Abstract

Modern deep recommender models are trained under a continual learning paradigm, relying on massive and continuously growing streaming behavioral logs. In large-scale platforms, retraining models on full historical data for architecture comparison or iteration is prohibitively expensive, severely slowing down model development. This challenge calls for data-efficient approaches that can faithfully approximate full-data training behavior without repeatedly processing the entire evolving data stream. We formulate this problem as \emph{streaming dataset distillation for recommender systems} and propose \textbf{DIET}, a unified framework that maintains a compact distilled dataset which evolves alongside streaming data while preserving training-critical signals. Unlike existing dataset distillation methods that construct a static distilled set, DIET models distilled data as an evolving training memory and updates it in a stage-wise manner to remain aligned with long-term training dynamics. DIET enables effective continual distillation through principled initialization from influential samples and selective updates guided by influence-aware memory addressing within a bi-level optimization framework. Experiments on large-scale recommendation benchmarks demonstrate that DIET compresses training data to as little as \textbf{1-2\%} of the original size while preserving performance trends consistent with full-data training, reducing model iteration cost by up to \textbf{60$\times$}. Moreover, the distilled datasets produced by DIET generalize well across different model architectures, highlighting streaming dataset distillation as a scalable and reusable data foundation for recommender system development.