My Boli: Code-mixed Marathi-English Corpora, Pretrained Language Models and Evaluation Benchmarks

TL;DR

This paper introduces a large code-mixed Marathi-English corpus, pre-trained transformer models, and benchmark datasets for downstream tasks, significantly advancing research in low-resource code-mixed Indian languages.

Contribution

It provides the first dedicated code-mixed Marathi-English corpus, models, and evaluation benchmarks, filling a critical gap in low-resource language processing.

Findings

Models trained on the corpus outperform existing BERT models.

The datasets enable effective hate speech, sentiment, and language identification tasks.

All resources are publicly available for research use.

Abstract

The research on code-mixed data is limited due to the unavailability of dedicated code-mixed datasets and pre-trained language models. In this work, we focus on the low-resource Indian language Marathi which lacks any prior work in code-mixing. We present L3Cube-MeCorpus, a large code-mixed Marathi-English (Mr-En) corpus with 10 million social media sentences for pretraining. We also release L3Cube-MeBERT and MeRoBERTa, code-mixed BERT-based transformer models pre-trained on MeCorpus. Furthermore, for benchmarking, we present three supervised datasets MeHate, MeSent, and MeLID for downstream tasks like code-mixed Mr-En hate speech detection, sentiment analysis, and language identification respectively. These evaluation datasets individually consist of manually annotated \url{~}12,000 Marathi-English code-mixed tweets. Ablations show that the models trained on this novel corpus significantly outperform the existing state-of-the-art BERT models. This is the first work that presents artifacts for code-mixed Marathi research. All datasets and models are publicly released at https://github.com/l3cube-pune/MarathiNLP .