KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

TL;DR

KoSimpleQA is a challenging Korean factuality benchmark for LLMs, revealing current models' limitations and emphasizing the importance of reasoning capabilities for improved performance and abstention.

Contribution

The paper introduces KoSimpleQA, a new Korean factuality benchmark, and provides a comprehensive evaluation of LLMs, highlighting the importance of reasoning in factual QA.

Findings

Strongest models achieve only 33.7% accuracy on KoSimpleQA.

Performance rankings differ significantly from English benchmarks.

Reasoning capabilities improve models' knowledge elicitation and uncertainty abstention.

Abstract

We present $\textbf{Korean SimpleQA (KoSimpleQA)}$, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 1,000 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 33.7% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, our analysis of reasoning LLMs shows that engaging reasoning capabilities in the factual QA task can both help models better elicit their latent knowledge and improve their ability to abstain when uncertain. KoSimpleQA can be found at https://anonymous.4open.science/r/KoSimpleQA-62EB.