Beyond What to Select: A Plug-and-play Oscillatory Data-Volume Scheduling for Efficient Model Training

TL;DR

This paper introduces PODS, a dynamic data-volume scheduling framework that alternates data selection ratios during training to improve efficiency and generalization across multiple tasks.

Contribution

PODS is a lightweight, task-agnostic module that dynamically schedules data volume, enhancing training efficiency without sacrificing model performance.

Findings

Reduces ImageNet-1k training cost by 50%.

Speeds up LLM instruction tuning by over 2x.

Improves the efficiency-generalization trade-off across tasks.

Abstract

Data selection accelerates training by identifying representative training data while preserving model performance. However, existing methods mainly focus on designing sample-importance criteria, i.e., deciding what to select, while typically fixing the selected data volume as the target ratio throughout training. Thus, they are often dynamic in sample identity but static in data volume. In this work, we revisit data selection from an optimization perspective and show that selected-data training induces an implicit regularization effect modulated by the instantaneous selection ratio. This reveals a key trade-off: lower ratios amplify selection-induced regularization, whereas higher ratios preserve data coverage and optimization fidelity. Motivated by this insight, we propose PODS, a Plug-and-play Oscillatory Data-volume Scheduling framework. Rather than introducing another sample-scoring metric, PODS serves as a lightweight module that dynamically schedules how much data to select over training. Under the target selection ratio, PODS alternates between low-ratio regularization phases and high-ratio recovery phases to exploit selection-induced regularization without sacrificing optimization stability. With its lightweight, ratio-level, and task-agnostic design, PODS is compatible with existing static and dynamic selection methods and broadly applicable across training paradigms. Experiments across various datasets, architectures, and tasks show that PODS consistently improves the efficiency-generalization trade-off, e.g., reducing ImageNet-1k training cost by 50% with improved accuracy and accelerating LLM instruction tuning by over 2x without performance degradation.