Partial Forward Blocking: A Novel Data Pruning Paradigm for Lossless Training Acceleration

TL;DR

This paper introduces Partial Forward Blocking (PFB), a lossless data pruning method that accelerates training by selectively pruning less important samples based on shallow layer features, reducing computation without sacrificing accuracy.

Contribution

PFB is a novel adaptive pruning framework that assesses sample importance from shallow features, eliminating auxiliary computations and improving training efficiency.

Findings

Achieves 33% training time reduction on ImageNet.

Prunes 40% of data without accuracy loss.

Improves accuracy by 0.5% with pruning.

Abstract

The ever-growing size of training datasets enhances the generalization capability of modern machine learning models but also incurs exorbitant computational costs. Existing data pruning approaches aim to accelerate training by removing those less important samples. However, they often rely on gradients or proxy models, leading to prohibitive additional costs of gradient back-propagation and proxy model training. In this paper, we propose Partial Forward Blocking (PFB), a novel framework for lossless training acceleration. The efficiency of PFB stems from its unique adaptive pruning pipeline: sample importance is assessed based on features extracted from the shallow layers of the target model. Less important samples are then pruned, allowing only the retained ones to proceed with the subsequent forward pass and loss back-propagation. This mechanism significantly reduces the computational overhead of deep-layer forward passes and back-propagation for pruned samples, while also eliminating the need for auxiliary backward computations and proxy model training. Moreover, PFB introduces probability density as an indicator of sample importance. Combined with an adaptive distribution estimation module, our method dynamically prioritizes relatively rare samples, aligning with the constantly evolving training state. Extensive experiments demonstrate the significant superiority of PFB in performance and speed. On ImageNet, PFB achieves a 0.5% accuracy improvement and 33% training time reduction with 40% data pruned.