What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

TL;DR

This paper examines how compression techniques like pruning, distillation, and quantization affect small-data language models, demonstrating they can enhance efficiency and performance even under resource constraints.

Contribution

It is the first comprehensive study on applying compression methods to low-resource, small-data language models like AfriBERTa, revealing their benefits.

Findings

Compression improves efficiency of small-data models

Compression techniques enhance performance metrics beyond accuracy

Results align with effects observed in large models

Abstract

Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.