Language-driven Semantic Segmentation

TL;DR

LSeg is a new language-driven semantic segmentation model that uses text and image encoders to achieve zero-shot segmentation, generalizing to unseen categories without retraining.

Contribution

The paper introduces LSeg, a model that aligns pixel and text embeddings for zero-shot semantic segmentation, enabling flexible label handling and generalization to unseen classes.

Findings

Achieves competitive zero-shot segmentation performance.

Matches traditional segmentation accuracy with fixed labels.

Enables generalization to unseen categories without retraining.

Abstract

We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.