MTQ-Eval: Multilingual Text Quality Evaluation for Language Models

TL;DR

This paper introduces MTQ-Eval, a multilingual text quality evaluation framework leveraging large language models, trained on automatically generated preference data, showing improved performance across 115 languages and benefits for downstream tasks.

Contribution

The paper presents a novel multilingual evaluation framework, MTQ-Eval, trained on automatically generated quality preferences, extending LLM evaluation capabilities beyond specific tasks.

Findings

Improved evaluation performance across 115 languages.

Enhanced downstream task performance.

Effective use of automatically generated preference data.

Abstract

The use of large language models (LLMs) for evaluating outputs is becoming an increasingly effective and scalable approach. However, it remains uncertain whether this capability extends beyond task-specific evaluations to more general assessments of text quality, particularly in multilingual contexts. In this study, we introduce, MTQ-Eval, a novel framework for multilingual text quality evaluation that learns from examples of both high- and low-quality texts, adjusting its internal representations. To develop MTQ-Eval, we first automatically generate text quality preference data and then use it to train open-source base LLMs to align with ratings of high- and low-quality text. Our comprehensive evaluation across 115 languages demonstrates the improved performance of the proposed model. Upon further analysis, we find that this enhanced evaluation capability also leads to notable improvements in downstream tasks.