SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow

TL;DR

SemFlow introduces a unified ODE-based framework that jointly addresses semantic segmentation and image synthesis by modeling them as reverse problems, enabling reversible transfer between images and masks.

Contribution

It proposes a novel unified framework based on rectified flow theory, bridging semantic segmentation and image synthesis as symmetric reverse tasks.

Findings

Achieves competitive results on segmentation and synthesis tasks.

Enhances diversity of generated images without altering semantic categories.

Demonstrates the effectiveness of a reversible ODE-based approach.

Abstract

Semantic segmentation and semantic image synthesis are two representative tasks in visual perception and generation. While existing methods consider them as two distinct tasks, we propose a unified framework (SemFlow) and model them as a pair of reverse problems. Specifically, motivated by rectified flow theory, we train an ordinary differential equation (ODE) model to transport between the distributions of real images and semantic masks. As the training object is symmetric, samples belonging to the two distributions, images and semantic masks, can be effortlessly transferred reversibly. For semantic segmentation, our approach solves the contradiction between the randomness of diffusion outputs and the uniqueness of segmentation results. For image synthesis, we propose a finite perturbation approach to enhance the diversity of generated results without changing the semantic categories. Experiments show that our SemFlow achieves competitive results on semantic segmentation and semantic image synthesis tasks. We hope this simple framework will motivate people to rethink the unification of low-level and high-level vision.