SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

TL;DR

SToRe3D introduces a relevance-based sparsity framework for ViTs, enabling faster multi-view 3D detection with minimal accuracy loss, suitable for real-time applications.

Contribution

It presents a novel joint 2D-3D relevance-based sparsity method for ViTs, extending sparsity techniques to full-model and 3D object queries.

Findings

Achieves up to 3x faster inference on nuScenes with marginal accuracy loss.

Establishes real-time large-scale ViT-based 3D detection.

Maintains accuracy on planning-critical agents.

Abstract

Vision Transformers (ViTs) enable strong multi-view 3D detection but are limited by high inference latency from dense token and query processing across multiple views and large 3D regions. Existing sparsity methods, designed mainly for 2D vision, prune or merge image tokens but do not extend to full-model sparsity or address 3D object queries. We introduce SToRe3D, a relevance-aligned sparsity framework that jointly selects 2D image tokens and 3D object queries while storing filtered features for reactivation. Mutual 2D-3D relevance heads allocate compute to driving-critical content and preserve other embeddings. Evaluated on nuScenes and our new nuScenes-Relevance benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, establishing real-time large-scale ViT-based 3D detection while maintaining accuracy on planning-critical agents.