GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

TL;DR

GMMSeg introduces a generative semantic segmentation model that combines Gaussian Mixture Models with discriminative training, improving performance on both closed-set and open-world datasets by capturing class-conditional distributions.

Contribution

The paper proposes GMMSeg, a novel segmentation approach that integrates generative GMMs with discriminative training, addressing out-of-distribution detection and enhancing segmentation accuracy.

Findings

Outperforms discriminative models on three closed-set datasets

Performs well on open-world datasets without modifications

Combines generative and discriminative strengths effectively

Abstract

Prevalent semantic segmentation solutions are, in essence, a dense discriminative classifier of p(class|pixel feature). Though straightforward, this de facto paradigm neglects the underlying data distribution p(pixel feature|class), and struggles to identify out-of-distribution data. Going beyond this, we propose GMMSeg, a new family of segmentation models that rely on a dense generative classifier for the joint distribution p(pixel feature,class). For each class, GMMSeg builds Gaussian Mixture Models (GMMs) via Expectation-Maximization (EM), so as to capture class-conditional densities. Meanwhile, the deep dense representation is end-to-end trained in a discriminative manner, i.e., maximizing p(class|pixel feature). This endows GMMSeg with the strengths of both generative and discriminative models. With a variety of segmentation architectures and backbones, GMMSeg outperforms the discriminative counterparts on three closed-set datasets. More impressively, without any modification, GMMSeg even performs well on open-world datasets. We believe this work brings fundamental insights into the related fields.