L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT

TL;DR

This paper introduces L3Cube-IndicSBERT, a simple method to convert multilingual BERT into effective cross-lingual sentence representations, especially for Indian languages, outperforming existing models on similarity tasks.

Contribution

Proposes a straightforward fine-tuning approach using synthetic datasets to create high-quality multilingual sentence embeddings for Indic languages and beyond.

Findings

IndicSBERT outperforms LaBSE, LASER, and MPNet on Indic language similarity tasks.

The approach works effectively for non-Indic languages like German and French.

Monolingual SBERT models perform competitively with IndicSBERT.

Abstract

The multilingual Sentence-BERT (SBERT) models map different languages to common representation space and are useful for cross-language similarity and mining tasks. We propose a simple yet effective approach to convert vanilla multilingual BERT models into multilingual sentence BERT models using synthetic corpus. We simply aggregate translated NLI or STS datasets of the low-resource target languages together and perform SBERT-like fine-tuning of the vanilla multilingual BERT model. We show that multilingual BERT models are inherent cross-lingual learners and this simple baseline fine-tuning approach without explicit cross-lingual training yields exceptional cross-lingual properties. We show the efficacy of our approach on 10 major Indic languages and also show the applicability of our approach to non-Indic languages German and French. Using this approach, we further present L3Cube-IndicSBERT, the first multilingual sentence representation model specifically for Indian languages Hindi, Marathi, Kannada, Telugu, Malayalam, Tamil, Gujarati, Odia, Bengali, and Punjabi. The IndicSBERT exhibits strong cross-lingual capabilities and performs significantly better than alternatives like LaBSE, LASER, and paraphrase-multilingual-mpnet-base-v2 on Indic cross-lingual and monolingual sentence similarity tasks. We also release monolingual SBERT models for each of the languages and show that IndicSBERT performs competitively with its monolingual counterparts. These models have been evaluated using embedding similarity scores and classification accuracy.