Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition

TL;DR

This paper introduces TLS-Assist, a modular system that enhances LLM-based autonomous driving by explicitly recognizing traffic lights and signs, leading to improved safety and driving performance in simulated environments.

Contribution

We propose TLS-Assist, a plug-and-play, model-agnostic module that improves LLM autonomous driving agents by explicitly detecting and incorporating traffic light and sign information.

Findings

Up to 14% improvement in driving performance.

Significant reduction in traffic infractions.

Effective in both single-view and multi-view setups.

Abstract

Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving, showing promising reasoning capabilities and potential to generalize across diverse traffic situations. However, current LLM-based driving agents lack explicit mechanisms to enforce traffic rules and often struggle to reliably detect small, safety-critical objects such as traffic lights and signs. To address this limitation, we introduce TLS-Assist, a modular redundancy layer that augments LLM-based autonomous driving agents with explicit traffic light and sign recognition. TLS-Assist converts detections into structured natural language messages that are injected into the LLM input, enforcing explicit attention to safety-critical cues. The framework is plug-and-play, model-agnostic, and supports both single-view and multi-view camera setups. We evaluate TLS-Assist in a closed-loop setup on the LangAuto benchmark in CARLA. The results demonstrate relative driving performance improvements of up to 14% over LMDrive and 7% over BEVDriver, while consistently reducing traffic light and sign infractions. We publicly release the code and models on https://github.com/iis-esslingen/TLS-Assist.