Multiscale Video Transformers for Class Agnostic Segmentation in Autonomous Driving

TL;DR

This paper introduces a multiscale video transformer that efficiently performs class-agnostic segmentation in autonomous driving, accurately detecting unknown objects using motion cues without relying on optical flow.

Contribution

It proposes a novel end-to-end trainable video transformer with a memory-centric design and multiscale query decoding, improving efficiency and accuracy over existing methods.

Findings

Outperforms multiscale baselines on DAVIS'16, KITTI, and Cityscapes datasets.

Maintains high-resolution spatiotemporal features with shared memory.

Demonstrates real-time, robust dense prediction suitable for safety-critical robotics.

Abstract

Ensuring safety in autonomous driving is a complex challenge requiring handling unknown objects and unforeseen driving scenarios. We develop multiscale video transformers capable of detecting unknown objects using only motion cues. Video semantic and panoptic segmentation often relies on known classes seen during training, overlooking novel categories. Recent visual grounding with large language models is computationally expensive, especially for pixel-level output. We propose an efficient video transformer trained end-to-end for class-agnostic segmentation without optical flow. Our method uses multi-stage multiscale query-memory decoding and a scale-specific random drop-token to ensure efficiency and accuracy, maintaining detailed spatiotemporal features with a shared, learnable memory module. Unlike conventional decoders that compress features, our memory-centric design preserves high-resolution information at multiple scales. We evaluate on DAVIS'16, KITTI, and Cityscapes. Our method consistently outperforms multiscale baselines while being efficient in GPU memory and run-time, demonstrating a promising direction for real-time, robust dense prediction in safety-critical robotics.