LoCar: Localization-Aware Evaluation of In-Vehicle Assistants through Fine-Grained Sociolinguistic Control

TL;DR

This paper introduces a new evaluation framework for in-vehicle conversational AI, emphasizing Korean localization and highlighting challenges in honorific control and strategic interaction metrics.

Contribution

It presents a domain-specific evaluation method for in-vehicle assistants, focusing on linguistic accuracy and interaction reliability, especially for Korean language models.

Findings

Korean honorific control in LLMs is unstable.

Models perform poorly on clarification and proactivity metrics.

Evaluation framework emphasizes reliability and linguistic precision.

Abstract

While Large Language Models (LLMs) are increasingly integrated into in-vehicle conversational systems, identifying the optimal model remains challenging due to the lack of domain-specific evaluation standards tailored to real-world deployment requirements. In this paper, we propose a novel evaluation framework for in-vehicle assistants, with a particular focus on Korean-language localization. Our empirical analysis reveals notable patterns in model behavior. First, fine-grained Korean honorific control remains unstable in current LLMs, indicating that precise speech-level realization must be explicitly evaluated in localization settings. Second, models exhibit weaker performance in strategic conversational metrics like clarification and proactivity. Our analysis suggests this stems from the inherent subjective complexity of these tasks, where our framework adopts a conservative evaluation stance to prioritize reliability. Together, our findings underscore that automotive AI must move beyond general competence toward precise linguistic tailoring and reliable, safety-oriented interaction management.