Average Is Not Enough: Caveats of Multilingual Evaluation

TL;DR

This paper highlights the limitations of using average performance metrics in multilingual evaluation, emphasizing the need for qualitative linguistic analysis and visualization to detect biases favoring dominant language families.

Contribution

It introduces a methodology combining qualitative linguistic analysis with visualization tools to identify biases in multilingual evaluation results.

Findings

Published results can be linguistically biased.

Visualization with URIEL database detects evaluation biases.

Qualitative analysis is essential for fair multilingual assessment.

Abstract

This position paper discusses the problem of multilingual evaluation. Using simple statistics, such as average language performance, might inject linguistic biases in favor of dominant language families into evaluation methodology. We argue that a qualitative analysis informed by comparative linguistics is needed for multilingual results to detect this kind of bias. We show in our case study that results in published works can indeed be linguistically biased and we demonstrate that visualization based on URIEL typological database can detect it.