L3Cube-IndicNews: News-based Short Text and Long Document Classification Datasets in Indic Languages

TL;DR

This paper introduces L3Cube-IndicNews, a comprehensive multilingual dataset for classifying news headlines and articles in 10 Indian languages, enabling improved NLP models and cross-lingual analysis.

Contribution

It provides a high-quality, multi-length news classification dataset for 10 Indic languages, facilitating research and development of language-specific and cross-lingual models.

Findings

Evaluated with 4 models including monolingual BERT and IndicSBERT

Achieved promising classification performance across languages

Shared datasets and models publicly for community use

Abstract

In this work, we introduce L3Cube-IndicNews, a multilingual text classification corpus aimed at curating a high-quality dataset for Indian regional languages, with a specific focus on news headlines and articles. We have centered our work on 10 prominent Indic languages, including Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada, Odia, Malayalam, and Punjabi. Each of these news datasets comprises 10 or more classes of news articles. L3Cube-IndicNews offers 3 distinct datasets tailored to handle different document lengths that are classified as: Short Headlines Classification (SHC) dataset containing the news headline and news category, Long Document Classification (LDC) dataset containing the whole news article and the news category, and Long Paragraph Classification (LPC) containing sub-articles of the news and the news category. We maintain consistent labeling across all 3 datasets for in-depth length-based analysis. We evaluate each of these Indic language datasets using 4 different models including monolingual BERT, multilingual Indic Sentence BERT (IndicSBERT), and IndicBERT. This research contributes significantly to expanding the pool of available text classification datasets and also makes it possible to develop topic classification models for Indian regional languages. This also serves as an excellent resource for cross-lingual analysis owing to the high overlap of labels among languages. The datasets and models are shared publicly at https://github.com/l3cube-pune/indic-nlp