LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models

TL;DR

LaoBench is a comprehensive benchmark designed to evaluate large language models' understanding and reasoning in Lao, addressing the lack of evaluation resources for low-resource Southeast Asian languages.

Contribution

It introduces the first large-scale, multidimensional Lao language benchmark with expert-curated samples and a hybrid pipeline for high-quality evaluation.

Findings

State-of-the-art LLMs lag behind human experts in Lao understanding.

Models perform poorly in culturally grounded reasoning and translation fidelity.

LaoBench enables fair black-box evaluation with secure, held-out subsets.

Abstract

The rapid advancement of large language models (LLMs) has not been matched by their evaluation in low-resource languages, especially Southeast Asian languages like Lao. To fill this gap, we introduce \textbf{LaoBench}, the first large-scale, high-quality, and multidimensional benchmark for assessing LLM language understanding and reasoning in Lao. LaoBench contains \textbf{17,000+} expert-curated samples across three dimensions: culturally grounded knowledge application, curriculum-aligned K12 education, and bilingual translation among Lao, Chinese, and English. It includes open-source and held-out subsets, where the held-out portion enables secure black-box evaluation via a controlled service to improve fairness and data security. We construct LaoBench with a hybrid pipeline that combines expert authoring with agent-assisted verification, ensuring linguistic accuracy, cultural relevance, and educational validity. We evaluate diverse state-of-the-art open-source and closed-source LLMs, and find that even strong multilingual models lag behind human experts, particularly in culturally grounded reasoning and translation fidelity. We hope LaoBench will catalyze research on Lao and other underrepresented Southeast Asian languages for more inclusive multilingual evaluation.