Autoregressive Universal Video Segmentation Model

TL;DR

AUSM is a unified autoregressive model for both prompted and unprompted video segmentation, capable of handling arbitrary-length streams efficiently and outperforming prior methods on standard benchmarks.

Contribution

The paper introduces AUSM, a novel architecture that unifies prompted and unprompted video segmentation using sequential mask prediction inspired by language modeling.

Findings

Outperforms prior universal streaming video segmentation methods.

Achieves up to 2.5x faster training on 16-frame sequences.

Maintains fixed-size spatial state for arbitrary-length videos.

Abstract

Recent video foundation models such as SAM2 excel at prompted video segmentation by treating masks as a general-purpose primitive. However, many real-world settings require unprompted segmentation that aims to detect and track all objects in a video without external cues, leaving today's landscape fragmented across task-specific models and pipelines. We recast streaming video segmentation as sequential mask prediction, analogous to language modeling, and introduce the Autoregressive Universal Segmentation Model (AUSM), a single architecture that unifies both prompted and unprompted video segmentation. Built on recent state-space models, AUSM maintains a fixed-size spatial state and scales to video streams of arbitrary length. Furthermore, all components of AUSM are designed for parallel training across frames, yielding substantial speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS 2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior universal streaming video segmentation methods and achieves up to 2.5x faster training on 16-frame sequences.