Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering

TL;DR

KoLasSimpleQA is a comprehensive multilingual benchmark designed to evaluate the factual knowledge and self-awareness of Large Language Models across nine languages and two domains, highlighting performance gaps and guiding future improvements.

Contribution

This paper introduces KoLasSimpleQA, the first multilingual benchmark for factual ability of LLMs, covering multiple languages and domains for comprehensive evaluation.

Findings

Significant performance differences between general and language-specific domains.

Mainstream LLMs show varied accuracy and robustness across languages.

Benchmark facilitates targeted evaluation and model optimization in multilingual settings.

Abstract

We introduce KoLasSimpleQA, the first benchmark evaluating the multilingual factual ability of Large Language Models (LLMs). Inspired by existing research, we created the question set with features such as single knowledge point coverage, absolute objectivity, unique answers, and temporal stability. These questions enable efficient evaluation using the LLM-as-judge paradigm, testing both the LLMs' factual memory and self-awareness ("know what they don't know"). KoLasSimpleQA expands existing research in two key dimensions: (1) Breadth (Multilingual Coverage): It includes 9 languages, supporting global applicability evaluation. (2) Depth (Dual Domain Design): It covers both the general domain (global facts) and the language-specific domain (such as history, culture, and regional traditions) for a comprehensive assessment of multilingual capabilities. We evaluated mainstream LLMs, including traditional LLM and emerging Large Reasoning Models. Results show significant performance differences between the two domains, particularly in performance metrics, ranking, calibration, and robustness. This highlights the need for targeted evaluation and optimization in multilingual contexts. We hope KoLasSimpleQA will help the research community better identify LLM capability boundaries in multilingual contexts and provide guidance for model optimization. We will release KoLasSimpleQA at https://github.com/opendatalab/KoLasSimpleQA .