niksss at HinglishEval: Language-agnostic BERT-based Contextual Embeddings with Catboost for Quality Evaluation of the Low-Resource Synthetically Generated Code-Mixed Hinglish Text

TL;DR

This paper presents a BERT-based approach using sentence embeddings and Catboost classifier for evaluating the quality of synthetic code-mixed Hinglish text, achieving top rankings in a shared task.

Contribution

Introduces a language-agnostic BERT-based embedding method combined with Catboost for quality assessment of low-resource code-mixed text, winning multiple subtask rankings.

Findings

Achieved 1st place in subtask B

Achieved 3rd place in subtask A

Demonstrated effectiveness of sentence embeddings for quality prediction

Abstract

This paper describes the system description for the HinglishEval challenge at INLG 2022. The goal of this task was to investigate the factors influencing the quality of the code-mixed text generation system. The task was divided into two subtasks, quality rating prediction and annotators disagreement prediction of the synthetic Hinglish dataset. We attempted to solve these tasks using sentence-level embeddings, which are obtained from mean pooling the contextualized word embeddings for all input tokens in our text. We experimented with various classifiers on top of the embeddings produced for respective tasks. Our best-performing system ranked 1st on subtask B and 3rd on subtask A.