When, Why, and Which Pretrained GANs Are Useful?

TL;DR

This paper analyzes the mechanisms and practical considerations of finetuning pretrained GANs, revealing that initialization influences coverage more than sample fidelity and providing guidelines for selecting suitable checkpoints.

Contribution

It offers an in-depth analysis of how pretrained GANs influence finetuning, clarifies the roles of generator and discriminator, and proposes a practical checkpoint selection method.

Findings

Pretraining affects model coverage more than sample fidelity.

Pretrained generator and discriminator roles are explicitly characterized.

A simple recipe for selecting GAN checkpoints for finetuning is proposed.

Abstract

The literature has proposed several methods to finetune pretrained GANs on new datasets, which typically results in higher performance compared to training from scratch, especially in the limited-data regime. However, despite the apparent empirical benefits of GAN pretraining, its inner mechanisms were not analyzed in-depth, and understanding of its role is not entirely clear. Moreover, the essential practical details, e.g., selecting a proper pretrained GAN checkpoint, currently do not have rigorous grounding and are typically determined by trial and error. This work aims to dissect the process of GAN finetuning. First, we show that initializing the GAN training process by a pretrained checkpoint primarily affects the model's coverage rather than the fidelity of individual samples. Second, we explicitly describe how pretrained generators and discriminators contribute to the finetuning process and explain the previous evidence on the importance of pretraining both of them. Finally, as an immediate practical benefit of our analysis, we describe a simple recipe to choose an appropriate GAN checkpoint that is the most suitable for finetuning to a particular target task. Importantly, for most of the target tasks, Imagenet-pretrained GAN, despite having poor visual quality, appears to be an excellent starting point for finetuning, resembling the typical pretraining scenario of discriminative computer vision models.