BanglaLlama: LLaMA for Bangla Language

TL;DR

BanglaLlama introduces new Bangla instruction datasets and language models to improve NLP performance for the 5th largest spoken language, addressing resource scarcity.

Contribution

The paper presents two large Bangla instruction datasets and a family of Bangla-specific LLMs, establishing new benchmarks for Bangla language processing.

Findings

Datasets improve model performance on benchmarks

BanglaLlama models outperform existing models on Bangla tasks

Open-source resources set new standards for Bangla NLP

Abstract

Bangla is a language spoken by approximately 240 million native speakers and around 300 million people worldwide. Despite being the 5th largest spoken language in the world, Bangla is still a "low-resource" language, and existing pretrained language models often struggle to perform well on Bangla Language Processing (BLP) tasks. This paper addresses this gap by: (1) introducing two high-quality translated Bangla-instruction datasets totaling 224k samples - Bangla-Orca (172k) and Bangla-Alpaca (52k); and (2) leveraging these datasets to develop BanglaLlama, an open-source family of Bangla-specific LLMs, consisting of five base and instruct variants. We present our methodology, two large datasets, and comprehensive benchmarking results showcasing the effectiveness of our dataset and model on multiple benchmarks. We believe our proposed datasets and models will serve as the new standard baseline for future research focused on this widely spoken yet "low-resource" language.