The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models

TL;DR

GaoYao is a comprehensive benchmark designed to evaluate multilingual and multicultural abilities of large language models across 26 languages and 51 nations, addressing limitations of previous benchmarks with deep cultural analysis and expert localization.

Contribution

The paper introduces GaoYao, a new benchmark with extensive cultural and linguistic coverage, a unified evaluation framework, and in-depth diagnostic analysis of LLMs' cultural competencies.

Findings

Significant geographical performance disparities in LLMs.

Gaps identified between different evaluation tasks.

Expert localization improves benchmark quality and coverage.

Abstract

Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao).