Grounding Driving VLA via Inverse Kinematics

TL;DR

This paper introduces a novel inverse kinematics-inspired approach to improve visual grounding in driving trajectory models, enabling smaller models to perform comparably to much larger ones.

Contribution

It re-designs driving VLAs with a new visual prediction objective and an inverse kinematics network, significantly enhancing visual grounding and trajectory planning performance.

Findings

A 0.5B model matches larger models' performance on benchmarks.

Visual grounding improves notably in dynamic driving scenarios.

The approach suppresses reliance on ego status and textual shortcuts.

Abstract

Existing Driving VLAs predict trajectories while largely ignoring their visual tokens -- a phenomenon we trace not to insufficient training but to a structurally ill-posed task formulation. We show that trajectory recovery, when viewed through the lens of inverse kinematics, requires both a current and a future visual state as boundary conditions; existing VLAs supply only the former, which encourages the model to shortcut through ego status and text commands alone. To address this, we re-design Driving VLA in the style of an inverse kinematics solver. First, a next visual state prediction objective that requires the LLM to predict the future visual scene provides dense visual supervision and suppresses shortcut paths. Second, a separate Inverse Kinematics Network (a cross-attention-based conditional diffusion model) that takes only the current and future visual states as input is designed to suppress reliance on ego status and textual shortcuts during trajectory decoding. With this simple prescription alone, our 0.5B-scale model recovers visual grounding and reaches trajectory planning performance comparable to 7B--8B VLAs more than an order of magnitude larger, on both the closed-loop NAVSIM-v2 and the nuScenes benchmarks. Extensive analysis further shows that this improvement stems from a recovered ability to exploit visual features, with the effect being most pronounced in dynamic driving situations such as turning.