RESPOND: Risk-Enhanced Structured Pattern for LLM-driven Online Node-level Decision-making

TL;DR

RESPOND introduces a structured, risk-aware pattern framework for LLM-driven driving agents, significantly improving scene retrieval, safety, and adaptability in high-risk scenarios through explicit spatial risk encoding and reflection mechanisms.

Contribution

This work presents RESPOND, a novel structured decision-making framework that enhances LLM-based driving agents with explicit risk patterns, hybrid decision pipelines, and reflection for improved safety and personalization.

Findings

Outperforms state-of-the-art LLM and reinforcement learning agents in highway-env with fewer collisions.

Reduces risk in 84.9% of high-risk cut-in scenarios from the HighD dataset.

Enables rapid, safe action reuse and personalized driving style adaptation.

Abstract

Current LLM-based driving agents that rely on unstructured plain-text memory suffer from low-precision scene retrieval and inefficient reflection. To address this limitation, we present RESPOND, a structured decision-making framework for LLM-driven agents grounded in explicit risk patterns. RESPOND represents each ego-centric scene using a unified 5 by 3 matrix that encodes spatial topology and road constraints, enabling consistent and reliable retrieval of spatial risk configurations. Based on this representation, a hybrid rule and LLM decision pipeline is developed with a two-tier memory mechanism. In high-risk contexts, exact pattern matching enables rapid and safe reuse of verified actions, while in low-risk contexts, sub-pattern matching supports personalized driving style adaptation. In addition, a pattern-aware reflection mechanism abstracts tactical corrections from crash and near-miss frames to update structured memory, achieving one-crash-to-generalize learning. Extensive experiments demonstrate the effectiveness of RESPOND. In highway-env, RESPOND outperforms state-of-the-art LLM-based and reinforcement learning based driving agents while producing substantially fewer collisions. With step-wise human feedback, the agent acquires a Sporty driving style within approximately 20 decision steps through sub-pattern abstraction. For real-world validation, RESPOND is evaluated on 53 high-risk cut-in scenarios extracted from the HighD dataset. For each event, intervention is applied immediately before the cut-in and RESPOND re-decides the driving action. Compared to recorded human behavior, RESPOND reduces subsequent risk in 84.9 percent of scenarios, demonstrating its practical feasibility under real-world driving conditions. These results highlight RESPONDs potential for autonomous driving, personalized driving assistance, and proactive hazard mitigation.