KnowDiffuser: A Knowledge-Guided Diffusion Planner with LLM Reasoning

TL;DR

KnowDiffuser integrates language model semantic reasoning with diffusion models to generate physically feasible, context-aware trajectories for autonomous driving, improving planning accuracy and interpretability.

Contribution

It introduces a novel framework combining language models and diffusion models for motion planning, enhancing semantic understanding and trajectory generation in autonomous driving.

Findings

Outperforms existing planners on the nuPlan benchmark.

Effectively bridges semantic understanding and physical trajectory generation.

Demonstrates robustness in open-loop and closed-loop evaluations.

Abstract

Recent advancements in Language Models (LMs) have demonstrated strong semantic reasoning capabilities, enabling their application in high-level decision-making for autonomous driving (AD). However, LMs operate over discrete token spaces and lack the ability to generate continuous, physically feasible trajectories required for motion planning. Meanwhile, diffusion models have proven effective at generating reliable and dynamically consistent trajectories, but often lack semantic interpretability and alignment with scene-level understanding. To address these limitations, we propose \textbf{KnowDiffuser}, a knowledge-guided motion planning framework that tightly integrates the semantic understanding of language models with the generative power of diffusion models. The framework employs a language model to infer context-aware meta-actions from structured scene representations, which are then mapped to prior trajectories that anchor the subsequent denoising process. A two-stage truncated denoising mechanism refines these trajectories efficiently, preserving both semantic alignment and physical feasibility. Experiments on the nuPlan benchmark demonstrate that KnowDiffuser significantly outperforms existing planners in both open-loop and closed-loop evaluations, establishing a robust and interpretable framework that effectively bridges the semantic-to-physical gap in AD systems.