Reasoning About Traversability: Language-Guided Off-Road 3D Trajectory Planning

TL;DR

This paper introduces a novel framework for off-road trajectory planning that uses language refinement and terrain-aware optimization to improve autonomous driving in unstructured environments.

Contribution

It presents a new language refinement method and terrain-aware preference optimization to enhance scene understanding and trajectory accuracy in off-road driving scenarios.

Findings

Trajectory error reduced from 1.01m to 0.97m

Traversability compliance improved from 0.621 to 0.644

Elevation inconsistency decreased from 0.428 to 0.322

Abstract

While Vision-Language Models (VLMs) enable high-level semantic reasoning for end-to-end autonomous driving, particularly in unstructured environments, existing off-road datasets suffer from language annotations that are weakly aligned with vehicle actions and terrain geometry. To address this misalignment, we propose a language refinement framework that restructures annotations into action-aligned pairs, enabling a VLM to generate refined scene descriptions and 3D future trajectories directly from a single image. To further encourage terrain-aware planning, we introduce a preference optimization strategy that constructs geometry-aware hard negatives and explicitly penalizes trajectories inconsistent with local elevation profiles. Furthermore, we propose off-road-specific metrics to quantify traversability compliance and elevation consistency, addressing the limitations of conventional on-road evaluation. Experiments on the ORAD-3D benchmark demonstrate that our approach reduces average trajectory error from 1.01m to 0.97m, improves traversability compliance from 0.621 to 0.644, and decreases elevation inconsistency from 0.428 to 0.322, highlighting the efficacy of action-aligned supervision and terrain-aware optimization for robust off-road driving.