Efficient NLP Model Finetuning via Multistage Data Filtering

TL;DR

This paper introduces a multistage data filtering approach for NLP model finetuning that significantly reduces training data and time with minimal accuracy loss by selectively skipping training passes based on importance metrics.

Contribution

The paper presents a novel streaming data filtering method with loss thresholding and meta prediction to improve finetuning efficiency, compatible with existing techniques.

Findings

Reduces training data by up to 5.3 times.

Speeds up training by up to 6.8 times.

Maintains accuracy with minor degradation.

Abstract

As model finetuning is central to the modern NLP, we set to maximize its efficiency. Motivated by redundancy in training examples and the sheer sizes of pretrained models, we exploit a key opportunity: training only on important data. To this end, we set to filter training examples in a streaming fashion, in tandem with training the target model. Our key techniques are two: (1) automatically determine a training loss threshold for skipping backward training passes; (2) run a meta predictor for further skipping forward training passes. We integrate the above techniques in a holistic, three-stage training process. On a diverse set of benchmarks, our method reduces the required training examples by up to 5.3$\times$ and training time by up to 6.8$\times$, while only seeing minor accuracy degradation. Our method is effective even when training one epoch, where each training example is encountered only once. It is simple to implement and is compatible with the existing finetuning techniques. Code is available at: https://github.com/xo28/efficient- NLP-multistage-training