Enhancing End-to-End Autonomous Driving with Risk Semantic Distillaion from VLM

TL;DR

This paper introduces Risk Semantic Distillation (RSD), a novel framework that uses Vision-Language Models to improve the generalization and risk awareness of end-to-end autonomous driving systems, leading to better handling of complex scenarios.

Contribution

The paper proposes RSD, a new method that distills risk estimates from VLMs into BEV features, enhancing interpretability and generalization in autonomous driving models.

Findings

RSD improves perception and planning in complex driving scenarios.

Enhanced BEV features lead to better risk attention and handling of risky objects.

Significant performance gains on the Bench2Drive benchmark.

Abstract

The autonomous driving (AD) system has exhibited remarkable performance in complex driving scenarios. However, generalization is still a key limitation for the current system, which refers to the ability to handle unseen scenarios or unfamiliar sensor configurations.Related works have explored the use of Vision-Language Models (VLMs) to address few-shot or zero-shot tasks. While promising, these methods introduce a new challenge: the emergence of a hybrid AD system, where two distinct systems are used to plan a trajectory, leading to potential inconsistencies. Alternative research directions have explored Vision-Language-Action (VLA) frameworks that generate control actions from VLM directly. However, these end-to-end solutions demonstrate prohibitive computational demands. To overcome these challenges, we introduce Risk Semantic Distillation (RSD), a novel framework that leverages VLMs to enhance the training of End-to-End (E2E) AD backbones. By providing risk attention for key objects, RSD addresses the issue of generalization. Specifically, we introduce RiskHead, a plug-in module that distills causal risk estimates from Vision-Language Models into Bird's-Eye-View (BEV) features, yielding interpretable risk-attention maps.This approach allows BEV features to learn richer and more nuanced risk attention representations, which directly enhance the model's ability to handle spatial boundaries and risky objects.By focusing on risk attention, RSD aligns better with human-like driving behavior, which is essential to navigate in complex and dynamic environments. Our experiments on the Bench2Drive benchmark demonstrate the effectiveness of RSD in managing complex and unpredictable driving conditions. Due to the enhanced BEV representations enabled by RSD, we observed a significant improvement in both perception and planning capabilities.