Self-Ensembling GAN for Cross-Domain Semantic Segmentation

TL;DR

This paper introduces SE-GAN, a self-ensembling GAN framework that leverages cross-domain data to improve semantic segmentation performance while reducing annotation requirements, with theoretical analysis and extensive experiments confirming its effectiveness.

Contribution

It proposes a novel self-ensembling GAN architecture for semi-supervised semantic segmentation, enhancing stability and performance over existing methods.

Findings

SE-GAN significantly outperforms state-of-the-art approaches.

Theoretical analysis shows an O(1/√N) generalization bound.

Using a simple discriminator network improves generalizability.

Abstract

Deep neural networks (DNNs) have greatly contributed to the performance gains in semantic segmentation. Nevertheless, training DNNs generally requires large amounts of pixel-level labeled data, which is expensive and time-consuming to collect in practice. To mitigate the annotation burden, this paper proposes a self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain data for semantic segmentation. In SE-GAN, a teacher network and a student network constitute a self-ensembling model for generating semantic segmentation maps, which together with a discriminator, forms a GAN. Despite its simplicity, we find SE-GAN can significantly boost the performance of adversarial training and enhance the stability of the model, the latter of which is a common barrier shared by most adversarial training-based methods. We theoretically analyze SE-GAN and provide an $\mathcal O(1/\sqrt{N})$ generalization bound ($N$ is the training sample size), which suggests controlling the discriminator's hypothesis complexity to enhance the generalizability. Accordingly, we choose a simple network as the discriminator. Extensive and systematic experiments in two standard settings demonstrate that the proposed method significantly outperforms current state-of-the-art approaches. The source code of our model is available online (https://github.com/YonghaoXu/SE-GAN).