GAN-Supervised Dense Visual Alignment

TL;DR

GAN-Supervised Learning introduces a novel framework that jointly trains discriminative models and GAN-generated data for dense visual alignment, outperforming existing methods without requiring labeled data.

Contribution

The paper presents GANgealing, a new end-to-end algorithm for dense visual alignment that leverages GANs and outperforms prior supervised and self-supervised approaches.

Findings

Successfully aligns complex data across eight datasets.

Outperforms previous self-supervised algorithms.

Matches or exceeds state-of-the-art supervised methods, sometimes by 3x.

Abstract

We propose GAN-Supervised Learning, a framework for learning discriminative models and their GAN-generated training data jointly end-to-end. We apply our framework to the dense visual alignment problem. Inspired by the classic Congealing method, our GANgealing algorithm trains a Spatial Transformer to map random samples from a GAN trained on unaligned data to a common, jointly-learned target mode. We show results on eight datasets, all of which demonstrate our method successfully aligns complex data and discovers dense correspondences. GANgealing significantly outperforms past self-supervised correspondence algorithms and performs on-par with (and sometimes exceeds) state-of-the-art supervised correspondence algorithms on several datasets -- without making use of any correspondence supervision or data augmentation and despite being trained exclusively on GAN-generated data. For precise correspondence, we improve upon state-of-the-art supervised methods by as much as $3\times$. We show applications of our method for augmented reality, image editing and automated pre-processing of image datasets for downstream GAN training.