Towards Building Efficient Sentence BERT Models using Layer Pruning

TL;DR

This paper explores layer pruning in Sentence BERT models to create smaller, efficient models that maintain high embedding quality, outperforming similarly sized models trained from scratch.

Contribution

It demonstrates that layer pruning effectively reduces model size and complexity while preserving embedding performance, outperforming comparable scratch-trained models.

Findings

Pruned models perform competitively with full models.

Pruned models outperform similarly sized scratch-trained models.

Layer pruning reduces computational complexity significantly.

Abstract

This study examines the effectiveness of layer pruning in creating efficient Sentence BERT (SBERT) models. Our goal is to create smaller sentence embedding models that reduce complexity while maintaining strong embedding similarity. We assess BERT models like Muril and MahaBERT-v2 before and after pruning, comparing them with smaller, scratch-trained models like MahaBERT-Small and MahaBERT-Smaller. Through a two-phase SBERT fine-tuning process involving Natural Language Inference (NLI) and Semantic Textual Similarity (STS), we evaluate the impact of layer reduction on embedding quality. Our findings show that pruned models, despite fewer layers, perform competitively with fully layered versions. Moreover, pruned models consistently outperform similarly sized, scratch-trained models, establishing layer pruning as an effective strategy for creating smaller, efficient embedding models. These results highlight layer pruning as a practical approach for reducing computational demand while preserving high-quality embeddings, making SBERT models more accessible for languages with limited technological resources.