Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

TL;DR

This paper introduces X-Value, a comprehensive benchmark for evaluating large language models' ability to judge deep-level values across multiple languages, addressing a critical gap in multilingual AI evaluation.

Contribution

It presents a novel two-stage human-AI annotation framework and the first cross-lingual values judgment benchmark, X-Value, covering 14 languages and 7 global issue categories.

Findings

LLMs show limitations in cross-lingual values judgment accuracy.

Performance varies significantly across languages and issue categories.

Current models need improvement in values-aware content evaluation.

Abstract

As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on factual task performance, neglecting the ability to judge content's deep-level values across multiple languages. To bridge this gap, we first reveal two primary challenges in constructing values judgment benchmarks, cultural diversity and disciplinary complexity, and propose a novel two-stage human-AI collaborative annotation framework to alleviate them. This framework identifies the issue scope and nature, establishes specific annotation criteria, and utilizes multiple LLMs for final review. Building upon this framework, we introduce \textbf{X-Value}, the first \textit{Cross-lingual Values Judgment Benchmark} designed to evaluate the capability of LLMs in judging deep-level values of content. X-Value comprises 4,750 Question-Answer pairs across 14 languages, covering 7 major global issue categories, and provides 12 granular annotation metadata to facilitate a rigorous evaluation of model performance. Systematic evaluations of X-Value are conducted across 17 LLMs using distinct prompting strategies. Multi-dimensional analysis of accuracy and F1-scores reveals their limitations in cross-lingual values judgment and indicates performance disparities across categories and languages. This work highlights the urgent need to improve the underlying, values-aware content judgment capability of LLMs.\footnote{Samples of X-Value are available at https://huggingface.co/datasets/Whitolf/X-Value.}