PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders

TL;DR

The paper introduces PMT, a fast, Transformer-based segmentation model that operates on frozen Vision Foundation Model features, achieving high accuracy with significantly reduced latency for both image and video segmentation tasks.

Contribution

It proposes the Plain Mask Transformer (PMT), a novel decoder that enables fast, encoder-free segmentation using frozen VFM features, combining simplicity, speed, and multi-task capability.

Findings

PMT matches state-of-the-art frozen-encoder performance on image segmentation.

PMT is up to 3x faster than existing methods on image tasks.

For video segmentation, PMT is up to 8x faster and on par with fully finetuned models.

Abstract

Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.