Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation

TL;DR

This paper introduces DIA-REFINE, an iterative framework guiding large language models to produce faithful Korean dialect translations, addressing dialect gaps and evaluation issues with new metrics and feedback loops.

Contribution

The paper proposes DIA-REFINE, a novel iterative refinement framework with new metrics for dialect translation, improving faithfulness and evaluation accuracy in Korean dialect translation tasks.

Findings

DIA-REFINE improves dialect fidelity across experiments.

New metrics distinguish true dialect translation from false positives.

In-context examples enhance dialect translation performance.

Abstract

Standard-to-dialect machine translation remains challenging due to a persistent dialect gap in large language models and evaluation distortions inherent in n-gram metrics, which favor source copying over authentic dialect translation. In this paper, we propose the dialect refinement (DIA-REFINE) framework, which guides LLMs toward faithful target dialect outputs through an iterative loop of translation, verification, and feedback using external dialect classifiers. To address the limitations of n-gram-based metrics, we introduce the dialect fidelity score (DFS) to quantify linguistic shift and the target dialect ratio (TDR) to measure the success of dialect translation. Experiments on Korean dialects across zero-shot and in-context learning baselines demonstrate that DIA-REFINE consistently enhances dialect fidelity. The proposed metrics distinguish between False Success cases, where high n-gram scores obscure failures in dialectal translation, and True Attempt cases, where genuine attempts at dialectal translation yield low n-gram scores. We also observed that models exhibit varying degrees of responsiveness to the framework, and that integrating in-context examples further improves the translation of dialectal expressions. Our work establishes a robust framework for goal-directed, inclusive dialect translation, providing both rigorous evaluation and critical insights into model performance.