AD-EE: Early Exiting for Fast and Reliable Vision-Language Models in Autonomous Driving

TL;DR

This paper introduces AD-EE, an early exit framework for vision-language models in autonomous driving, significantly reducing latency and improving accuracy by intelligently determining when to stop processing.

Contribution

The paper presents a novel early exit method tailored for autonomous driving VLMs, utilizing domain-specific cues and causal inference to optimize inference efficiency.

Findings

Latency reduced by up to 57.58%

Object detection accuracy improved by up to 44%

Effective across multiple VLM architectures

Abstract

With the rapid advancement of autonomous driving, deploying Vision-Language Models (VLMs) to enhance perception and decision-making has become increasingly common. However, the real-time application of VLMs is hindered by high latency and computational overhead, limiting their effectiveness in time-critical driving scenarios. This challenge is particularly evident when VLMs exhibit over-inference, continuing to process unnecessary layers even after confident predictions have been reached. To address this inefficiency, we propose AD-EE, an Early Exit framework that incorporates domain characteristics of autonomous driving and leverages causal inference to identify optimal exit layers. We evaluate our method on large-scale real-world autonomous driving datasets, including Waymo and the corner-case-focused CODA, as well as on a real vehicle running the Autoware Universe platform. Extensive experiments across multiple VLMs show that our method significantly reduces latency, with maximum improvements reaching up to 57.58%, and enhances object detection accuracy, with maximum gains of up to 44%.