On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages

TL;DR

This paper investigates layer pruning in BERT models for low-resource languages, showing that pruned models can match the performance of larger models while reducing complexity and computational costs.

Contribution

It demonstrates that layer pruning, especially from the middle, effectively reduces BERT model size without sacrificing accuracy in low-resource language tasks.

Findings

Pruned models perform comparably to full models.

Middle-layer pruning is most effective.

Monolingual BERT outperforms multilingual models.

Abstract

This study explores the effectiveness of layer pruning for developing more efficient BERT models tailored to specific downstream tasks in low-resource languages. Our primary objective is to evaluate whether pruned BERT models can maintain high performance while reducing model size and complexity. We experiment with several BERT variants, including MahaBERT-v2 and Google-Muril, applying different pruning strategies and comparing their performance to smaller, scratch-trained models like MahaBERT-Small and MahaBERT-Smaller. We fine-tune these models on Marathi datasets, specifically Short Headlines Classification (SHC), Long Paragraph Classification (LPC) and Long Document Classification (LDC), to assess their classification accuracy. Our findings demonstrate that pruned models, despite having fewer layers, achieve comparable performance to their fully-layered counterparts while consistently outperforming scratch-trained models of similar size. Notably, pruning layers from the middle of the model proves to be the most effective strategy, offering performance competitive with pruning from the top and bottom. However, there is no clear winner, as different pruning strategies perform better in different model and dataset combinations. Additionally, monolingual BERT models outperform multilingual ones in these experiments. This approach, which reduces computational demands, provides a faster and more efficient alternative to training smaller models from scratch, making advanced NLP models more accessible for low-resource languages without compromising classification accuracy.