Mind the (optimality) Gap: A Gap-Aware Learning Rate Scheduler for Adversarial Nets

TL;DR

This paper introduces a novel adaptive learning rate scheduler for adversarial nets that maintains a balanced training process, reduces tuning efforts, and improves model quality in tasks like image generation and domain adaptation.

Contribution

The paper proposes a gap-aware learning rate scheduler that dynamically adjusts the adversary's learning rate based on the loss of an ideal adversarial network, enhancing training stability and performance.

Findings

Scheduler reduces training divergence in adversarial nets.

Requires less hyperparameter tuning, saving time and resources.

Achieves up to 27% improvement in image quality and 3% in domain accuracy.

Abstract

Adversarial nets have proved to be powerful in various domains including generative modeling (GANs), transfer learning, and fairness. However, successfully training adversarial nets using first-order methods remains a major challenge. Typically, careful choices of the learning rates are needed to maintain the delicate balance between the competing networks. In this paper, we design a novel learning rate scheduler that dynamically adapts the learning rate of the adversary to maintain the right balance. The scheduler is driven by the fact that the loss of an ideal adversarial net is a constant known a priori. The scheduler is thus designed to keep the loss of the optimized adversarial net close to that of an ideal network. We run large-scale experiments to study the effectiveness of the scheduler on two popular applications: GANs for image generation and adversarial nets for domain adaptation. Our experiments indicate that adversarial nets trained with the scheduler are less likely to diverge and require significantly less tuning. For example, on CelebA, a GAN with the scheduler requires only one-tenth of the tuning budget needed without a scheduler. Moreover, the scheduler leads to statistically significant improvements in model quality, reaching up to $27\%$ in Frechet Inception Distance for image generation and $3\%$ in test accuracy for domain adaptation.