Vacaspati: A Diverse Corpus of Bangla Literature

TL;DR

Vacaspati is a comprehensive Bangla literature corpus that enables improved NLP models, including a lightweight Electra model, demonstrating better performance on downstream tasks compared to models from other corpora.

Contribution

The paper introduces Vacaspati, a diverse and large-scale Bangla literature corpus, and develops efficient NLP models trained on it, advancing Bangla language processing.

Findings

Vacaspati contains over 11 million sentences and 115 million words.

Vac-BERT performs better or similarly to larger models on downstream tasks.

Models trained on Vacaspati outperform those from other corpora.

Abstract

Bangla (or Bengali) is the fifth most spoken language globally; yet, the state-of-the-art NLP in Bangla is lagging for even simple tasks such as lemmatization, POS tagging, etc. This is partly due to lack of a varied quality corpus. To alleviate this need, we build Vacaspati, a diverse corpus of Bangla literature. The literary works are collected from various websites; only those works that are publicly available without copyright violations or restrictions are collected. We believe that published literature captures the features of a language much better than newspapers, blogs or social media posts which tend to follow only a certain literary pattern and, therefore, miss out on language variety. Our corpus Vacaspati is varied from multiple aspects, including type of composition, topic, author, time, space, etc. It contains more than 11 million sentences and 115 million words. We also built a word embedding model, Vac-FT, using FastText from Vacaspati as well as trained an Electra model, Vac-BERT, using the corpus. Vac-BERT has far fewer parameters and requires only a fraction of resources compared to other state-of-the-art transformer models and yet performs either better or similar on various downstream tasks. On multiple downstream tasks, Vac-FT outperforms other FastText-based models. We also demonstrate the efficacy of Vacaspati as a corpus by showing that similar models built from other corpora are not as effective. The models are available at https://bangla.iitk.ac.in/.