Make Your ViT-based Multi-view 3D Detectors Faster via Token Compression

TL;DR

This paper introduces TokenCompression3D, a method that accelerates ViT-based multi-view 3D detectors by intelligently compressing tokens, maintaining high accuracy while significantly improving inference speed for real-time applications.

Contribution

The paper proposes a novel token compression approach for ViT backbones in 3D detection, enhancing efficiency without sacrificing detection performance.

Findings

Achieves up to 30% inference speedup on nuScenes dataset.

Maintains near state-of-the-art detection accuracy.

Effective scaling with larger ViT models and input resolutions.

Abstract

Slow inference speed is one of the most crucial concerns for deploying multi-view 3D detectors to tasks with high real-time requirements like autonomous driving. Although many sparse query-based methods have already attempted to improve the efficiency of 3D detectors, they neglect to consider the backbone, especially when using Vision Transformers (ViT) for better performance. To tackle this problem, we explore the efficient ViT backbones for multi-view 3D detection via token compression and propose a simple yet effective method called TokenCompression3D (ToC3D). By leveraging history object queries as foreground priors of high quality, modeling 3D motion information in them, and interacting them with image tokens through the attention mechanism, ToC3D can effectively determine the magnitude of information densities of image tokens and segment the salient foreground tokens. With the introduced dynamic router design, ToC3D can weigh more computing resources to important foreground tokens while compressing the information loss, leading to a more efficient ViT-based multi-view 3D detector. Extensive results on the large-scale nuScenes dataset show that our method can nearly maintain the performance of recent SOTA with up to 30% inference speedup, and the improvements are consistent after scaling up the ViT and input resolution. The code will be made at https://github.com/DYZhang09/ToC3D.