JU_NLP at HinglishEval: Quality Evaluation of the Low-Resource Code-Mixed Hinglish Text

TL;DR

This paper presents a neural network-based system for evaluating the quality of low-resource, synthetically generated code-mixed Hinglish text, focusing on rating and disagreement prediction tasks.

Contribution

It introduces a Bi-LSTM model utilizing multilingual embeddings and one-hot encodings for quality assessment of Hinglish text, a low-resource language setting.

Findings

Achieved F1 score of 0.11 in rating prediction

Achieved F1 score of 0.18 in disagreement prediction

Reported mean squared errors of 6.0 and 5.0 respectively

Abstract

In this paper we describe a system submitted to the INLG 2022 Generation Challenge (GenChal) on Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text. We implement a Bi-LSTM-based neural network model to predict the Average rating score and Disagreement score of the synthetic Hinglish dataset. In our models, we used word embeddings for English and Hindi data, and one hot encodings for Hinglish data. We achieved a F1 score of 0.11, and mean squared error of 6.0 in the average rating score prediction task. In the task of Disagreement score prediction, we achieve a F1 score of 0.18, and mean squared error of 5.0.