K-MetBench: A Multi-Dimensional Benchmark for Fine-Grained Evaluation of Expert Reasoning, Locality, and Multimodality in Meteorology

TL;DR

K-MetBench is a comprehensive benchmark designed to evaluate Korean weather AI models across visual reasoning, logical validity, cultural understanding, and domain-specific analysis, revealing significant modality and reasoning gaps.

Contribution

This paper introduces K-MetBench, a multidimensional, expert-level evaluation framework for Korean weather models, highlighting cultural and modality challenges in AI reasoning.

Findings

Models show a significant modality gap in interpreting diagrams.

Models hallucinate logic despite correct predictions.

Korean models outperform larger global models in local contexts.

Abstract

The development of practical (multimodal) large language model assistants for Korean weather forecasters is hindered by the absence of a multidimensional, expert-level evaluation framework grounded in authoritative sources. To address this, we introduce K-MetBench, a diagnostic benchmark grounded in national qualification exams. It exposes critical gaps across four dimensions: expert visual reasoning of charts, logical validity via expert-verified rationales, Korean-specific geo-cultural comprehension, and fine-grained domain analysis. Our evaluation of 55 models reveals a profound modality gap in interpreting specialized diagrams and a reasoning gap where models hallucinate logic despite correct predictions. Crucially, Korean models outperform significantly larger global models in local contexts, demonstrating that parameter scaling alone cannot resolve cultural dependencies. K-MetBench serves as a roadmap for developing reliable, culturally aware expert AI agents. The dataset is available at https://huggingface.co/datasets/soyeonbot/K-MetBench .