Pragmatic Competence Evaluation of Large Language Models for the Korean Language

TL;DR

This study evaluates Korean LLMs' pragmatic understanding using both automatic and human assessments, revealing strengths and limitations in their ability to interpret context-dependent expressions and highlighting areas for improvement.

Contribution

It introduces a pragmatic evaluation framework for Korean LLMs, combining MCQs and OEQs, and analyzes how prompting techniques affect pragmatic inference capabilities.

Findings

GPT-4 scores highest in pragmatic understanding

Few-shot learning improves performance

Chain-of-Thought prompting may limit pragmatic inference

Abstract

Benchmarks play a significant role in the current evaluation of Large Language Models (LLMs), yet they often overlook the models' abilities to capture the nuances of human language, primarily focusing on evaluating embedded knowledge and technical skills. To address this gap, our study evaluates how well LLMs understand context-dependent expressions from a pragmatic standpoint, specifically in Korean. We use both Multiple-Choice Questions (MCQs) for automatic evaluation and Open-Ended Questions (OEQs) assessed by human experts. Our results show that GPT-4 leads with scores of 81.11 in MCQs and 85.69 in OEQs, closely followed by HyperCLOVA X. Additionally, while few-shot learning generally improves performance, Chain-of-Thought (CoT) prompting tends to encourage literal interpretations, which may limit effective pragmatic inference. Our findings highlight the need for LLMs to better understand and generate language that reflects human communicative norms.