CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

TL;DR

CoordGAN introduces a novel GAN architecture that explicitly learns dense pixel-level correspondences across images by disentangling structure and texture, enabling applications like segmentation transfer and improved interpretability.

Contribution

This work presents CoordGAN, a structure-texture disentangled GAN that explicitly learns dense correspondences and improves structure and texture disentanglement over prior methods.

Findings

Successfully extracts dense correspondence maps for generated and real images.

Achieves better structure and texture disentanglement compared to existing approaches.

Demonstrates effective segmentation mask transfer across datasets.

Abstract

Recent advances show that Generative Adversarial Networks (GANs) can synthesize images with smooth variations along semantically meaningful latent directions, such as pose, expression, layout, etc. While this indicates that GANs implicitly learn pixel-level correspondences across images, few studies explored how to extract them explicitly. In this work, we introduce Coordinate GAN (CoordGAN), a structure-texture disentangled GAN that learns a dense correspondence map for each generated image. We represent the correspondence maps of different images as warped coordinate frames transformed from a canonical coordinate frame, i.e., the correspondence map, which describes the structure (e.g., the shape of a face), is controlled via a transformation. Hence, finding correspondences boils down to locating the same coordinate in different correspondence maps. In CoordGAN, we sample a transformation to represent the structure of a synthesized instance, while an independent texture branch is responsible for rendering appearance details orthogonal to the structure. Our approach can also extract dense correspondence maps for real images by adding an encoder on top of the generator. We quantitatively demonstrate the quality of the learned dense correspondences through segmentation mask transfer on multiple datasets. We also show that the proposed generator achieves better structure and texture disentanglement compared to existing approaches. Project page: https://jitengmu.github.io/CoordGAN/