A Gamified Evaluation and Recruitment Platform for Low Resource Language Machine Translation Systems

TL;DR

This paper proposes a gamified platform to recruit and evaluate human evaluators for low-resource language machine translation, addressing dataset and evaluator scarcity issues in developing more nuanced assessment methods.

Contribution

It introduces a novel recruitment and evaluation platform designed specifically for low-resource languages, combining gamification to improve evaluator engagement and resource availability.

Findings

Designed a recruitment platform for low-resource language evaluators

Proposed gamification to enhance evaluator participation

Discussed potential applications in NLP research

Abstract

Human evaluators provide necessary contributions in evaluating large language models. In the context of Machine Translation (MT) systems for low-resource languages (LRLs), this is made even more apparent since popular automated metrics tend to be string-based, and therefore do not provide a full picture of the nuances of the behavior of the system. Human evaluators, when equipped with the necessary expertise of the language, will be able to test for adequacy, fluency, and other important metrics. However, the low resource nature of the language means that both datasets and evaluators are in short supply. This presents the following conundrum: How can developers of MT systems for these LRLs find adequate human evaluators and datasets? This paper first presents a comprehensive review of existing evaluation procedures, with the objective of producing a design proposal for a platform that addresses the resource gap in terms of datasets and evaluators in developing MT systems. The result is a design for a recruitment and gamified evaluation platform for developers of MT systems. Challenges are also discussed in terms of evaluating this platform, as well as its possible applications in the wider scope of Natural Language Processing (NLP) research.