Bridging the Data Gap: Creating a Hindi Text Summarization Dataset from the English XSUM

TL;DR

This paper presents an automated, cost-effective method to create a high-quality Hindi text summarization dataset by translating and adapting from the English XSUM dataset, validated with advanced evaluation techniques.

Contribution

Introduces a scalable framework for generating Hindi summarization datasets from English resources using translation and linguistic adaptation, validated with COMET and LLMs.

Findings

Created a diverse Hindi summarization dataset from XSUM

Validated dataset quality with COMET and LLMs

Facilitated Hindi NLP research with a cost-effective approach

Abstract

Current advancements in Natural Language Processing (NLP) have largely favored resource-rich languages, leaving a significant gap in high-quality datasets for low-resource languages like Hindi. This scarcity is particularly evident in text summarization, where the development of robust models is hindered by a lack of diverse, specialized corpora. To address this disparity, this study introduces a cost-effective, automated framework for creating a comprehensive Hindi text summarization dataset. By leveraging the English Extreme Summarization (XSUM) dataset as a source, we employ advanced translation and linguistic adaptation techniques. To ensure high fidelity and contextual relevance, we utilize the Crosslingual Optimized Metric for Evaluation of Translation (COMET) for validation, supplemented by the selective use of Large Language Models (LLMs) for curation. The resulting dataset provides a diverse, multi-thematic resource that mirrors the complexity of the original XSUM corpus. This initiative not only provides a direct tool for Hindi NLP research but also offers a scalable methodology for democratizing NLP in other underserved languages. By reducing the costs associated with dataset creation, this work fosters the development of more nuanced, culturally relevant models in computational linguistics.