Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding

TL;DR

This paper introduces CRS, a graph-based framework that combines geometric road structure with open-vocabulary semantics, improving structured reasoning in autonomous driving systems.

Contribution

CRS bridges the gap between semantic flexibility and structural precision by enabling joint execution of geometric and linguistic reasoning in a unified graph-grounded framework.

Findings

CRS enables automatic generation of complex question-answer pairs for road reasoning.

Training small models with CRS-enriched scenes improves compositional reasoning performance.

CRS-trained models show reduced failure modes, mainly limited to attribute recognition.

Abstract

Structured road understanding of lane geometry, topology, and traffic element relationships is foundational to safe autonomous driving. While vision-language models (VLMs) offer promising semantic flexibility, they lack the geometric and relational grounding required for precise road reasoning. Conversely, traditional modular systems, e.g., HD maps and topological road graphs, provide structural precision but remain semantically rigid. To bridge this gap, we introduce the Combined Road Substrate (CRS), a graph-grounded framework that makes geometric road structure and open-vocabulary semantics jointly executable in a single representation. CRS enables the automatic generation of compositionally complex and linguistically varied question-answer pairs via recursive graph queries, augmented with a "grounding for free" mechanism that ensures logical traceability to specific map elements, and procedurally extracted chain-of-thought supervision traces. We demonstrate that state-of-the-art VLMs - including large, closed-source models - struggle significantly with structured road reasoning, yet training a small 2- or 4-billion-parameter model with as few as 20 to 80 CRS-enriched scenes yields stable gains in compositional reasoning tasks of varying depth. Analysis of model behavior via verifiable reasoning traces reveals a systematic shift in failure modes: whereas baseline models fail at relational scene understanding, CRS-trained models reduce failures to attribute recognition, suggesting that the primary bottleneck in road understanding is not model scale, but the absence of structured supervision.