From Prompts to Pavement: LMMs-based Agentic Behavior-Tree Generation Framework for Autonomous Vehicles

TL;DR

This paper introduces an agentic framework using LLMs and multi-modal vision models to dynamically generate and adapt behavior trees for autonomous vehicles, enhancing safety and flexibility in unpredictable environments.

Contribution

It presents a novel LLM-based system that automatically creates and modifies behavior trees for AVs, reducing manual tuning and enabling real-time adaptation.

Findings

Successfully navigated around unexpected obstacles in simulation

Triggered only upon baseline BT failure, demonstrating adaptability

Extended static behavior trees to diverse scenarios with minimal intervention

Abstract

Autonomous vehicles (AVs) require adaptive behavior planners to navigate unpredictable, real-world environments safely. Traditional behavior trees (BTs) offer structured decision logic but are inherently static and demand labor-intensive manual tuning, limiting their applicability at SAE Level 5 autonomy. This paper presents an agentic framework that leverages large language models (LLMs) and multi-modal vision models (LVMs) to generate and adapt BTs on the fly. A specialized Descriptor agent applies chain-of-symbols prompting to assess scene criticality, a Planner agent constructs high-level sub-goals via in-context learning, and a Generator agent synthesizes executable BT sub-trees in XML format. Integrated into a CARLA+Nav2 simulation, our system triggers only upon baseline BT failure, demonstrating successful navigation around unexpected obstacles (e.g., street blockage) with no human intervention. Compared to a static BT baseline, this approach is a proof-of-concept that extends to diverse driving scenarios.