From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation

TL;DR

This paper introduces two expert-level Korean benchmarks, KMMLU-Redux and KMMLU-Pro, designed to evaluate large language models in academic and industrial contexts within Korea, emphasizing real-world applicability.

Contribution

The paper presents new Korean expert-level benchmarks, KMMLU-Redux and KMMLU-Pro, derived from national exams to better assess LLMs in professional and industrial domains.

Findings

Benchmarks effectively represent Korean industrial knowledge

Models show varying performance on these specialized benchmarks

Public dataset release facilitates further research

Abstract

The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applicability in real-world scenarios. In this paper, we introduce two Korean expert-level benchmarks. KMMLU-Redux, reconstructed from the existing KMMLU, consists of questions from the Korean National Technical Qualification exams, with critical errors removed to enhance reliability. KMMLU-Pro is based on Korean National Professional Licensure exams to reflect professional knowledge in Korea. Our experiments demonstrate that these benchmarks comprehensively represent industrial knowledge in Korea. We release our dataset publicly available.