Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception

TL;DR

Transtreaming introduces an adaptive delay-aware transformer for real-time streaming perception, enabling accurate, delay-compensated object detection across various devices, thus enhancing safety in autonomous systems.

Contribution

It proposes a novel delay-aware transformer that predicts multiple future frames and dynamically selects the best match, improving real-time detection accuracy and robustness.

Findings

Outperforms state-of-the-art methods in accuracy.

Operates effectively across diverse hardware platforms.

Meets real-time processing constraints on less powerful devices.

Abstract

Real-time object detection is critical for the decision-making process for many real-world applications, such as collision avoidance and path planning in autonomous driving. This work presents an innovative real-time streaming perception method, Transtreaming, which addresses the challenge of real-time object detection with dynamic computational delay. The core innovation of Transtreaming lies in its adaptive delay-aware transformer, which can concurrently predict multiple future frames and select the output that best matches the real-world present time, compensating for any system-induced computation delays. The proposed model outperforms the existing state-of-the-art methods, even in single-frame detection scenarios, by leveraging a transformer-based methodology. It demonstrates robust performance across a range of devices, from powerful V100 to modest 2080Ti, achieving the highest level of perceptual accuracy on all platforms. Unlike most state-of-the-art methods that struggle to complete computation within a single frame on less powerful devices, Transtreaming meets the stringent real-time processing requirements on all kinds of devices. The experimental results emphasize the system's adaptability and its potential to significantly improve the safety and reliability for many real-world systems, such as autonomous driving.