SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia
Chaoqun Liu, Wenxuan Zhang, Jiahao Ying, Mahani Aljunied, Anh Tuan, Luu, Lidong Bing

TL;DR
This paper presents SeaExam and SeaBench, two benchmarks based on real-world Southeast Asian scenarios to evaluate LLMs' multilingual and regional understanding, outperforming translated datasets in assessing performance.
Contribution
Introduction of SeaExam and SeaBench benchmarks based on authentic SEA data, improving evaluation of LLMs' regional language and application capabilities.
Findings
SeaExam and SeaBench better distinguish LLM performance on SEA tasks.
Real-world SEA data enhances benchmark effectiveness.
Benchmarks reflect practical regional language use.
Abstract
This study introduces two novel benchmarks, SeaExam and SeaBench, designed to evaluate the capabilities of Large Language Models (LLMs) in Southeast Asian (SEA) application scenarios. Unlike existing multilingual datasets primarily derived from English translations, these benchmarks are constructed based on real-world scenarios from SEA regions. SeaExam draws from regional educational exams to form a comprehensive dataset that encompasses subjects such as local history and literature. In contrast, SeaBench is crafted around multi-turn, open-ended tasks that reflect daily interactions within SEA communities. Our evaluations demonstrate that SeaExam and SeaBench more effectively discern LLM performance on SEA language tasks compared to their translated benchmarks. This highlights the importance of using real-world queries to assess the multilingual capabilities of LLMs.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Text Readability and Simplification · Second Language Acquisition and Learning
