ILID: Native Script Language Identification for Indian Languages

TL;DR

This paper introduces ILID, a comprehensive dataset and baseline models for identifying 23 Indian languages in noisy, short, and code-mixed texts, addressing a challenging NLP task with diverse linguistic features.

Contribution

The paper creates a large, annotated dataset of 250K sentences for 23 Indian languages and develops baseline models that outperform existing transformer-based approaches.

Findings

Baseline models outperform previous state-of-the-art transformers.

New dataset with 250K sentences for 23 Indian languages.

Models perform well in noisy and code-mixed environments.

Abstract

The language identification task is a crucial fundamental step in NLP. Often it serves as a pre-processing step for widely used NLP applications such as multilingual machine translation, information retrieval, question and answering, and text summarization. The core challenge of language identification lies in distinguishing languages in noisy, short, and code-mixed environments. This becomes even harder in case of diverse Indian languages that exhibit lexical and phonetic similarities, but have distinct differences. Many Indian languages share the same script, making the task even more challenging. Taking all these challenges into account, we develop and release a dataset of 250K sentences consisting of 23 languages including English and all 22 official Indian languages labeled with their language identifiers, where data in most languages are newly created. We also develop and release baseline models using state-of-the-art approaches in machine learning and fine-tuning pre-trained transformer models. Our models outperforms the state-of-the-art pre-trained transformer models for the language identification task. The dataset and the codes are available at https://yashingle-ai.github.io/ILID/ and in Huggingface open source libraries.