Multilingual Controlled Generation And Gold-Standard-Agnostic Evaluation of Code-Mixed Sentences

TL;DR

This paper introduces a novel controlled generation method for creating diverse code-mixed sentences from English and proposes GAME, a new evaluation metric that is language- and gold-standard-agnostic, reducing reliance on human annotations.

Contribution

It presents a controlled generation approach for code-mixed sentences and introduces GAME, a robust, annotation-free evaluation metric for code-mixing quality assessment.

Findings

GAME scores have lower standard deviation than BLEU.

The dataset includes gold-standard code-mixed sentences for four language pairs.

Controlled generation enables diverse, semantically equivalent code-mixed sentences.

Abstract

Code-mixing, the practice of alternating between two or more languages in an utterance, is a common phenomenon in multilingual communities. Due to the colloquial nature of code-mixing, there is no singular correct way to translate an English sentence into a code-mixed sentence. For this reason, standard n-gram-based MT evaluation metrics such as the BLEU score are not appropriate for code-mixed evaluation. To demonstrate this, we propose a novel method for code-mixed text generation: Controlled Generation, which parameterizes the code-mixing degree (CMD) and enables the generation of multiple semantically equivalent code-mixed sentences from a given English sentence. We introduce a robust new evaluation metric: GAME: A Gold-Standard Agnostic Measure for Evaluation of Code-Mixed Sentences. GAME is both language-agnostic and gold-standard-agnostic, i.e. unlike other metrics, GAME does not require gold-standard code-mixed sentences for evaluation, thus eliminating the need for human annotators in the code-mixed evaluation process. When used to evaluate semantically equivalent code-mixed sentences, we find that GAME scores have a lower standard deviation than BLEU scores. Further, we create and release a dataset containing gold-standard code-mixed sentences across 4 language pairs: English-{Hindi, Bengali, French, Spanish} to encourage more computational research on code-mixing.