HindiLLM: Large Language Model for Hindi

TL;DR

This paper introduces HindiLLM, a large language model for Hindi, trained through unsupervised pre-training and supervised fine-tuning, achieving superior performance on various language tasks compared to existing models.

Contribution

The paper presents the first large-scale Hindi language models, HindiLLM-Small and HindiLLM-Medium, with a new Hindi tokenizer and fine-tuning on multiple NLP tasks.

Findings

HindiLLM models outperform existing models on language tasks

Fine-tuned models achieve high accuracy in sentiment analysis and classification

A new high-quality Hindi text corpus and tokenizer are introduced.

Abstract

The advancements in the Large Language Model (LLM) have helped in solving several problems related to language processing. Most of the researches have focused on the English language only, because of its popularity and abundance on the internet. However, a high-performance language model for Hindi and other Indic languages is lacking in the literature. In this work, we have pre-trained two autoregressive LLM models for the Hindi language, namely HindiLLM-Small and HindiLLM-Medium. We use a two-step process comprising unsupervised pre-training and supervised fine-tuning. First, we create a large and high-quality text corpus for unsupervised pre-training. Next, we train a Byte-Pair Encoding, named HindiLLM tokenizer, using the pre-training text data. We then perform training on the unlabeled data, known as the pre-training step, to get the HindiLLM base models. Furthermore, we perform fine-tuning of the HindiLLM base models for different tasks like sentiment analysis, text classification, natural language inference, and multiple choice question-answer on popular labeled datasets to measure the real-world performance. The evaluation shows that the HindiLLM-based fine-tuned models outperform several models in most of the language related tasks.