See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers

TL;DR

See2Refine is a framework that uses vision-language models to automatically evaluate and improve LLM-generated eHMI actions for automated vehicles, enhancing communication without human supervision.

Contribution

It introduces a human-free, closed-loop system that refines LLM-based eHMI actions using automated visual feedback from VLMs, outperforming prompt-only and baseline methods.

Findings

Framework improves eHMI action appropriateness across modalities.

VLM evaluations align well with human preferences.

Refinement generalizes across different LLM sizes and modalities.

Abstract

Automated vehicles lack natural communication channels with other road users, making external Human-Machine Interfaces (eHMIs) essential for conveying intent and maintaining trust in shared environments. However, most eHMI studies rely on developer-crafted message-action pairs, which are difficult to adapt to diverse and dynamic traffic contexts. A promising alternative is to use Large Language Models (LLMs) as action designers that generate context-conditioned eHMI actions, yet such designers lack perceptual verification and typically depend on fixed prompts or costly human-annotated feedback for improvement. We present See2Refine, a human-free, closed-loop framework that uses vision-language model (VLM) perceptual evaluation as automated visual feedback to improve an LLM-based eHMI action designer. Given a driving context and a candidate eHMI action, the VLM evaluates the perceived appropriateness of the action, and this feedback is used to iteratively revise the designer's outputs, enabling systematic refinement without human supervision. We evaluate our framework across three eHMI modalities (lightbar, eyes, and arm) and multiple LLM model sizes. Across settings, our framework consistently outperforms prompt-only LLM designers and manually specified baselines in both VLM-based metrics and human-subject evaluations. Results further indicate that the improvements generalize across modalities and that VLM evaluations are well aligned with human preferences, supporting the robustness and effectiveness of See2Refine for scalable action design.