BanglaByT5: Byte-Level Modelling for Bangla

TL;DR

BanglaByT5 is a byte-level encoder-decoder model specifically designed for Bangla, demonstrating competitive performance and highlighting the advantages of byte-level modeling for morphologically rich languages.

Contribution

It introduces BanglaByT5, the first byte-level model tailored for Bangla, showing its effectiveness over traditional tokenization approaches.

Findings

Outperforms several multilingual and larger models in evaluations.

Proves byte-level modeling is effective for morphologically rich languages.

Serves as a lightweight, scalable tool for Bangla NLP applications.

Abstract

Large language models (LLMs) have achieved remarkable success across various natural language processing tasks. However, most LLM models use traditional tokenizers like BPE and SentencePiece, which fail to capture the finer nuances of a morphologically rich language like Bangla (Bengali). In this work, we introduce BanglaByT5, the first byte-level encoder-decoder model explicitly tailored for Bangla. Built upon a small variant of Googles ByT5 architecture, BanglaByT5 is pre-trained on a 14GB curated corpus combining high-quality literary and newspaper articles. Through zeroshot and supervised evaluations across generative and classification tasks, BanglaByT5 demonstrates competitive performance, surpassing several multilingual and larger models. Our findings highlight the efficacy of byte-level modelling for morphologically rich languages and highlight BanglaByT5 potential as a lightweight yet powerful tool for Bangla NLP, particularly in both resource-constrained and scalable environments.