HinGE: A Dataset for Generation and Evaluation of Code-Mixed Hinglish Text

TL;DR

This paper introduces HinGE, a new dataset of human and algorithm-generated Hinglish sentences, to support research in code-mixed text generation and evaluation, highlighting the limitations of existing metrics.

Contribution

The paper provides the first high-quality Hinglish dataset with human and rule-based generated sentences, addressing resource scarcity in code-mixed language research.

Findings

Widely-used evaluation metrics are ineffective on code-mixed data.

HinGE dataset enables better research in code-mixed language generation.

The dataset includes both human and algorithm-generated Hinglish sentences.

Abstract

Text generation is a highly active area of research in the computational linguistic community. The evaluation of the generated text is a challenging task and multiple theories and metrics have been proposed over the years. Unfortunately, text generation and evaluation are relatively understudied due to the scarcity of high-quality resources in code-mixed languages where the words and phrases from multiple languages are mixed in a single utterance of text and speech. To address this challenge, we present a corpus (HinGE) for a widely popular code-mixed language Hinglish (code-mixing of Hindi and English languages). HinGE has Hinglish sentences generated by humans as well as two rule-based algorithms corresponding to the parallel Hindi-English sentences. In addition, we demonstrate the inefficacy of widely-used evaluation metrics on the code-mixed data. The HinGE dataset will facilitate the progress of natural language generation research in code-mixed languages.