On Importance of Pruning and Distillation for Efficient Low Resource NLP

TL;DR

This paper investigates techniques like pruning and distillation to improve the efficiency of transformer models for low-resource languages, specifically Marathi, achieving significant speedups while maintaining accuracy.

Contribution

It introduces a combined approach of pruning and distillation tailored for Marathi NLP, demonstrating substantial efficiency gains with minimal accuracy loss.

Findings

25% pruning plus knowledge distillation yields 2.56x speedup

Efficiency techniques maintain baseline accuracy in Marathi NLP

Optimized models reduce environmental impact of NLP tasks

Abstract

The rise of large transformer models has revolutionized Natural Language Processing, leading to significant advances in tasks like text classification. However, this progress demands substantial computational resources, escalating training duration, and expenses with larger model sizes. Efforts have been made to downsize and accelerate English models (e.g., Distilbert, MobileBert). Yet, research in this area is scarce for low-resource languages. In this study, we explore the case of the low-resource Indic language Marathi. Leveraging the marathi-topic-all-doc-v2 model as our baseline, we implement optimization techniques to reduce computation time and memory usage. Our focus is on enhancing the efficiency of Marathi transformer models while maintaining top-tier accuracy and reducing computational demands. Using the MahaNews document classification dataset and the marathi-topic-all-doc-v2 model from L3Cube, we apply Block Movement Pruning, Knowledge Distillation, and Mixed Precision methods individually and in combination to boost efficiency. We demonstrate the importance of strategic pruning levels in achieving desired efficiency gains. Furthermore, we analyze the balance between efficiency improvements and environmental impact, highlighting how optimized model architectures can contribute to a more sustainable computational ecosystem. Implementing these techniques on a single GPU system, we determine that the optimal configuration is 25\% pruning + knowledge distillation. This approach yielded a 2.56x speedup in computation time while maintaining baseline accuracy levels.