HP-GAN: Harnessing pretrained networks for GAN improvement with FakeTwins and discriminator consistency

TL;DR

HP-GAN introduces a novel approach that leverages pretrained networks with self-supervised learning and discriminator consistency to significantly improve image quality and diversity in GANs across various datasets.

Contribution

The paper proposes HP-GAN, which incorporates FakeTwins and discriminator consistency to enhance GAN training using pretrained networks and self-supervised techniques.

Findings

Outperforms state-of-the-art in FID across 17 datasets

Generates more diverse and high-quality images

Effective in limited data scenarios

Abstract

Generative Adversarial Networks (GANs) have made significant progress in enhancing the quality of image synthesis. Recent methods frequently leverage pretrained networks to calculate perceptual losses or utilize pretrained feature spaces. In this paper, we extend the capabilities of pretrained networks by incorporating innovative self-supervised learning techniques and enforcing consistency between discriminators during GAN training. Our proposed method, named HP-GAN, effectively exploits neural network priors through two primary strategies: FakeTwins and discriminator consistency. FakeTwins leverages pretrained networks as encoders to compute a self-supervised loss and applies this through the generated images to train the generator, thereby enabling the generation of more diverse and high quality images. Additionally, we introduce a consistency mechanism between discriminators that evaluate feature maps extracted from Convolutional Neural Network (CNN) and Vision Transformer (ViT) feature networks. Discriminator consistency promotes coherent learning among discriminators and enhances training robustness by aligning their assessments of image quality. Our extensive evaluation across seventeen datasets-including scenarios with large, small, and limited data, and covering a variety of image domains-demonstrates that HP-GAN consistently outperforms current state-of-the-art methods in terms of Fr\'echet Inception Distance (FID), achieving significant improvements in image diversity and quality. Code is available at: https://github.com/higun2/HP-GAN.