BanglaEmbed: Efficient Sentence Embedding Models for a Low-Resource Language Using Cross-Lingual Distillation Techniques

TL;DR

This paper presents lightweight Bangla sentence embedding models using cross-lingual distillation from English transformers, improving performance on multiple NLP tasks for a low-resource language.

Contribution

Introduces novel cross-lingual knowledge distillation techniques to develop efficient Bangla sentence transformers for low-resource NLP applications.

Findings

Models outperform existing Bangla sentence transformers.

Lightweight architecture enables faster inference.

Effective across tasks like paraphrase detection and hate speech detection.

Abstract

Sentence-level embedding is essential for various tasks that require understanding natural language. Many studies have explored such embeddings for high-resource languages like English. However, low-resource languages like Bengali (a language spoken by almost two hundred and thirty million people) are still under-explored. This work introduces two lightweight sentence transformers for the Bangla language, leveraging a novel cross-lingual knowledge distillation approach. This method distills knowledge from a pre-trained, high-performing English sentence transformer. Proposed models are evaluated across multiple downstream tasks, including paraphrase detection, semantic textual similarity (STS), and Bangla hate speech detection. The new method consistently outperformed existing Bangla sentence transformers. Moreover, the lightweight architecture and shorter inference time make the models highly suitable for deployment in resource-constrained environments, making them valuable for practical NLP applications in low-resource languages.