Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving

TL;DR

Prune2Drive is a plug-and-play framework that efficiently prunes visual tokens in vision-language models for autonomous driving, significantly reducing computational costs while maintaining performance.

Contribution

It introduces a novel diversity-aware token selection and view-adaptive pruning method that requires no retraining, improving efficiency in multi-view autonomous driving models.

Findings

Achieves 6.40x speedup with only 3% performance drop

Reduces FLOPs to 13.4% of original

No retraining or attention map access needed

Abstract

Vision-Language Models (VLMs) have emerged as a promising paradigm in autonomous driving (AD), providing a unified framework for perception and decision-making. However, their real-world deployment is hindered by significant computational overhead when processing high-resolution, multi-view images. This complexity stems from the massive number of visual tokens, which increases inference latency and memory consumption due to the quadratic complexity of self-attention. To address these challenges, we propose Prune2Drive, a plug-and-play visual token pruning framework for multi-view VLMs in AD. Prune2Drive introduces two core innovations: (i) a diversity-aware token selection mechanism that prioritizes semantic and spatial coverage across views, and (ii) a view-adaptive pruning controller that automatically learns optimal pruning ratios based on camera importance to downstream tasks. Unlike prior methods, Prune2Drive requires no model retraining or access to attention maps, ensuring compatibility with modern efficient attention implementations. Extensive experiments on the DriveLM and DriveLMM-o1 benchmarks demonstrate that Prune2Drive achieves significant speedups and memory savings with minimal performance impact. When retaining only 10% of visual tokens, our method achieves a 6.40x speedup in the prefilling phase and consumes only 13.4% of the original FLOPs, with a mere 3% average performance drop on the DriveLM benchmark. Code is available at: https://github.com/MinhaoXiong/Prune2Drive.git