Multilingual CheckList: Generation and Evaluation

TL;DR

This paper proposes an automated method called TEA for generating multilingual evaluation CheckLists, enabling scalable and effective testing of language models across diverse languages with minimal human effort.

Contribution

The paper introduces TEA, an algorithm for automatically extracting CheckList templates from machine translations, reducing reliance on native speakers and scaling multilingual model evaluation.

Findings

TEA effectively extracts CheckLists with minimal human verification.

TEA-based CheckLists provide reliable estimates of model performance across languages.

Combining TEA with human verification balances cost, diversity, and accuracy.

Abstract

Multilingual evaluation benchmarks usually contain limited high-resource languages and do not test models for specific linguistic capabilities. CheckList is a template-based evaluation approach that tests models for specific capabilities. The CheckList template creation process requires native speakers, posing a challenge in scaling to hundreds of languages. In this work, we explore multiple approaches to generate Multilingual CheckLists. We device an algorithm - Template Extraction Algorithm (TEA) for automatically extracting target language CheckList templates from machine translated instances of a source language templates. We compare the TEA CheckLists with CheckLists created with different levels of human intervention. We further introduce metrics along the dimensions of cost, diversity, utility, and correctness to compare the CheckLists. We thoroughly analyze different approaches to creating CheckLists in Hindi. Furthermore, we experiment with 9 more different languages. We find that TEA followed by human verification is ideal for scaling Checklist-based evaluation to multiple languages while TEA gives a good estimates of model performance.