CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

TL;DR

CulturALL is a new benchmark designed to evaluate multilingual and multicultural grounded reasoning abilities of LLMs across diverse, real-world scenarios, revealing significant performance gaps.

Contribution

It introduces a comprehensive, human-AI collaboratively created benchmark with 2,610 challenging grounded task samples in 14 languages and 16 topics.

Findings

Best LLM achieves only 44.48% accuracy on CulturALL.

CulturALL covers diverse languages, regions, and topics.

The benchmark reveals substantial room for improvement in LLM grounded reasoning.

Abstract

Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks -- where models must reason within real-world, context-rich scenarios -- largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs' multilingual and multicultural competence on grounded tasks. CulturALL is built via a human--AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages from 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. Experiments show that the best LLM achieves 44.48% accuracy on CulturALL, underscoring substantial room for improvement.