Diversified Batch Selection for Training Acceleration

TL;DR

This paper introduces Diversified Batch Selection (DivBS), a reference-model-free method that efficiently selects diverse, representative samples to accelerate training without sacrificing performance, outperforming existing methods in various tasks.

Contribution

DivBS proposes a novel group-wise orthogonalized representativeness criterion for diversified batch selection, addressing redundancy and improving training efficiency.

Findings

DivBS achieves better performance-speedup trade-offs.

Extensive experiments validate DivBS's effectiveness across tasks.

DivBS outperforms existing reference-model-free methods.

Abstract

The remarkable success of modern machine learning models on large datasets often demands extensive training time and resource consumption. To save cost, a prevalent research line, known as online batch selection, explores selecting informative subsets during the training process. Although recent efforts achieve advancements by measuring the impact of each sample on generalization, their reliance on additional reference models inherently limits their practical applications, when there are no such ideal models available. On the other hand, the vanilla reference-model-free methods involve independently scoring and selecting data in a sample-wise manner, which sacrifices the diversity and induces the redundancy. To tackle this dilemma, we propose Diversified Batch Selection (DivBS), which is reference-model-free and can efficiently select diverse and representative samples. Specifically, we define a novel selection objective that measures the group-wise orthogonalized representativeness to combat the redundancy issue of previous sample-wise criteria, and provide a principled selection-efficient realization. Extensive experiments across various tasks demonstrate the significant superiority of DivBS in the performance-speedup trade-off. The code is publicly available.