Bridging Large-Model Reasoning and Real-Time Control via Agentic Fast-Slow Planning

TL;DR

This paper introduces a hierarchical framework that integrates large language models and classical control methods to improve autonomous driving robustness and efficiency across different timescales.

Contribution

It proposes Agentic Fast-Slow Planning, decoupling perception, reasoning, planning, and control, with novel bridges for semantic understanding and trajectory generation.

Findings

Reduces lateral deviation by up to 45% in CARLA simulations.

Decreases completion time by over 12% compared to baselines.

Enhances robustness under perturbations in autonomous driving scenarios.

Abstract

Large foundation models enable powerful reasoning for autonomous systems, but mapping semantic intent to reliable real-time control remains challenging. Existing approaches either (i) let Large Language Models (LLMs) generate trajectories directly - brittle, hard to verify, and latency-prone - or (ii) adjust Model Predictive Control (MPC) objectives online - mixing slow deliberation with fast control and blurring interfaces. We propose Agentic Fast-Slow Planning, a hierarchical framework that decouples perception, reasoning, planning, and control across natural timescales. The framework contains two bridges. Perception2Decision compresses scenes into ego-centric topologies using an on-vehicle Vision-Language Model (VLM) detector, then maps them to symbolic driving directives in the cloud with an LLM decision maker - reducing bandwidth and delay while preserving interpretability. Decision2Trajectory converts directives into executable paths: Semantic-Guided A* embeds language-derived soft costs into classical search to bias solutions toward feasible trajectories, while an Agentic Refinement Module adapts planner hyperparameters using feedback and memory. Finally, MPC tracks the trajectories in real time, with optional cloud-guided references for difficult cases. Experiments in CARLA show that Agentic Fast-Slow Planning improves robustness under perturbations, reducing lateral deviation by up to 45% and completion time by over 12% compared to pure MPC and an A*-guided MPC baseline. Code is available at https://github.com/cjychenjiayi/icra2026_AFSP.