Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives

TL;DR

BeyondDrive introduces a failure-aware imitation learning framework for autonomous driving that explicitly models safety-critical negative trajectories, improving safety and generalization over traditional methods.

Contribution

It proposes a novel negative trajectory generator, a diversity-aware sampling strategy, and a Replusive Distance Loss to enhance safety boundary modeling in imitation learning.

Findings

Achieves 89.7 PDMS on NAVSIMv1 benchmark, surpassing prior methods.

Effectively generalizes across different autonomous driving architectures.

Demonstrates strong zero-shot transferability on HUGSIM.

Abstract

Existing imitation learning methods for end-to-end autonomous driving predominantly learn from successful demonstrations by minimizing geometric deviations from expert trajectories. This paradigm implicitly assumes that spatial proximity implies behavioral safety, leading to a critical objective mismatch: trajectories with nearly identical imitation losses may exhibit drastically different safety outcomes, where one remains recoverable while the other results in collision. To address this limitation, we propose BeyondDrive, a failure-aware imitation learning framework that jointly learns from successful and failed driving behaviors. First, we introduce a flow matching-based negative trajectory generator that synthesizes safety-critical yet expert-proximate trajectories, enabling explicit modeling of safety asymmetry. Second, we develop a diversity-aware sampling strategy that mitigates mode collapse and improves coverage of diverse failure modes during negative trajectory generation. Third, we propose a Repulsive Distance Loss that simultaneously attracts predictions toward expert demonstrations while repelling them from hard negative trajectories, thereby establishing discriminative safety boundaries in trajectory space. Applied to the uni-modal baseline Latent TransFuser, BeyondDrive achieves 89.7 PDMS on the NAVSIMv1 closed-loop benchmark, outperforming prior state-of-the-art methods. Moreover, BeyondDrive generalizes effectively across different autonomous driving architectures, including multi-modal planners, and further demonstrates strong zero-shot transferability on the HUGSIM benchmark.