Real-time Stereo-based 3D Object Detection for Streaming Perception

TL;DR

This paper introduces StreamDSGN, a real-time stereo-based 3D object detection framework optimized for streaming perception in autonomous driving, improving accuracy by leveraging historical data and novel fusion strategies.

Contribution

The work presents the first real-time stereo-based 3D detection framework specifically designed for streaming perception, with innovative fusion and supervision techniques.

Findings

Significantly improves streaming average precision by up to 4.33%.

Effectively captures long-range spatial features with a large kernel backbone.

Enhances perception accuracy through feature fusion and motion supervision.

Abstract

The ability to promptly respond to environmental changes is crucial for the perception system of autonomous driving. Recently, a new task called streaming perception was proposed. It jointly evaluate the latency and accuracy into a single metric for video online perception. In this work, we introduce StreamDSGN, the first real-time stereo-based 3D object detection framework designed for streaming perception. StreamDSGN is an end-to-end framework that directly predicts the 3D properties of objects in the next moment by leveraging historical information, thereby alleviating the accuracy degradation of streaming perception. Further, StreamDSGN applies three strategies to enhance the perception accuracy: (1) A feature-flow-based fusion method, which generates a pseudo-next feature at the current moment to address the misalignment issue between feature and ground truth. (2) An extra regression loss for explicit supervision of object motion consistency in consecutive frames. (3) A large kernel backbone with a large receptive field for effectively capturing long-range spatial contextual features caused by changes in object positions. Experiments on the KITTI Tracking dataset show that, compared with the strong baseline, StreamDSGN significantly improves the streaming average precision by up to 4.33%. Our code is available at https://github.com/weiyangdaren/streamDSGN-pytorch.