SAM4D: Segment Anything in Camera and LiDAR Streams

TL;DR

SAM4D introduces a multi-modal foundation model for promptable segmentation in camera and LiDAR streams, utilizing novel alignment and memory mechanisms to improve robustness and efficiency in autonomous driving scene analysis.

Contribution

The paper presents SAM4D, a novel multi-modal, temporal segmentation model with unified encoding and motion-aware attention, plus an automated data engine for rapid pseudo-label generation.

Findings

Demonstrates strong cross-modal segmentation performance on Waymo-4DSeg

Achieves faster pseudo-label generation compared to manual annotation

Enhances temporal consistency in dynamic scenes

Abstract

We present SAM4D, a multi-modal and temporal foundation model designed for promptable segmentation across camera and LiDAR streams. Unified Multi-modal Positional Encoding (UMPE) is introduced to align camera and LiDAR features in a shared 3D space, enabling seamless cross-modal prompting and interaction. Additionally, we propose Motion-aware Cross-modal Memory Attention (MCMA), which leverages ego-motion compensation to enhance temporal consistency and long-horizon feature retrieval, ensuring robust segmentation across dynamically changing autonomous driving scenes. To avoid annotation bottlenecks, we develop a multi-modal automated data engine that synergizes VFM-driven video masklets, spatiotemporal 4D reconstruction, and cross-modal masklet fusion. This framework generates camera-LiDAR aligned pseudo-labels at a speed orders of magnitude faster than human annotation while preserving VFM-derived semantic fidelity in point cloud representations. We conduct extensive experiments on the constructed Waymo-4DSeg, which demonstrate the powerful cross-modal segmentation ability and great potential in data annotation of proposed SAM4D.