INT: Towards Infinite-frames 3D Detection with An Efficient Framework

TL;DR

This paper introduces INT, an efficient framework enabling near-infinite multi-frame 3D detection with minimal computational overhead, significantly boosting performance and achieving state-of-the-art results on large-scale datasets.

Contribution

The paper proposes a novel on-stream training and prediction framework that allows for infinite-frame 3D detection with constant computation, compatible with existing detectors.

Findings

7% performance boost on Waymo dataset

15% performance boost on nuScenes dataset

Achieves state-of-the-art on Waymo 3D Detection leaderboard

Abstract

It is natural to construct a multi-frame instead of a single-frame 3D detector for a continuous-time stream. Although increasing the number of frames might improve performance, previous multi-frame studies only used very limited frames to build their systems due to the dramatically increased computational and memory cost. To address these issues, we propose a novel on-stream training and prediction framework that, in theory, can employ an infinite number of frames while keeping the same amount of computation as a single-frame detector. This infinite framework (INT), which can be used with most existing detectors, is utilized, for example, on the popular CenterPoint, with significant latency reductions and performance improvements. We've also conducted extensive experiments on two large-scale datasets, nuScenes and Waymo Open Dataset, to demonstrate the scheme's effectiveness and efficiency. By employing INT on CenterPoint, we can get around 7% (Waymo) and 15% (nuScenes) performance boost with only 2~4ms latency overhead, and currently SOTA on the Waymo 3D Detection leaderboard.