GRAFT: Gradient-Aware Fast MaxVol Technique for Dynamic Data Sampling

TL;DR

GRAFT is a scalable, in-training data subset selection method that reduces computational costs and emissions in neural network training by dynamically selecting diverse, representative data samples using low-rank subspace techniques.

Contribution

GRAFT introduces a novel gradient-aware, low-rank subspace sampling method that improves training efficiency and reduces environmental impact without sacrificing accuracy.

Findings

GRAFT matches or exceeds baseline accuracy.

Reduces training time and energy consumption.

Lowers CO2 emissions during training.

Abstract

Training modern neural networks on large datasets is computationally and environmentally costly. We introduce GRAFT, a scalable in-training subset selection method that (i) extracts a low-rank feature representation for each batch, (ii) applies a Fast MaxVol sampler to select a small, diverse subset that spans the batch's dominant subspace, and (iii) dynamically adjusts the subset size using a gradient-approximation criterion. By operating in low-rank subspaces and training on carefully chosen examples instead of full batches, GRAFT preserves the training trajectory while reducing wall-clock time, energy consumption, and $\mathrm{CO}_2$ emissions. Across multiple benchmarks, GRAFT matches or exceeds recent selection baselines in both accuracy and efficiency, providing a favorable trade-off between accuracy, efficiency, and emissions.