Decoupled Learning for Conditional Adversarial Networks

TL;DR

This paper introduces decoupled learning for conditional adversarial networks, which separates the backpropagation of reconstruction and adversarial losses to improve stability and robustness in image generation.

Contribution

It proposes a novel network structure that disentangles loss backpropagation paths, reducing the need for manual loss balancing in adversarial training.

Findings

Demonstrates improved stability and robustness across experiments.

Validates effectiveness of decoupled learning in various settings.

Introduces a new evaluation metric, NRDS, for image quality assessment.

Abstract

Incorporating encoding-decoding nets with adversarial nets has been widely adopted in image generation tasks. We observe that the state-of-the-art achievements were obtained by carefully balancing the reconstruction loss and adversarial loss, and such balance shifts with different network structures, datasets, and training strategies. Empirical studies have demonstrated that an inappropriate weight between the two losses may cause instability, and it is tricky to search for the optimal setting, especially when lacking prior knowledge on the data and network. This paper gives the first attempt to relax the need of manual balancing by proposing the concept of \textit{decoupled learning}, where a novel network structure is designed that explicitly disentangles the backpropagation paths of the two losses. Experimental results demonstrate the effectiveness, robustness, and generality of the proposed method. The other contribution of the paper is the design of a new evaluation metric to measure the image quality of generative models. We propose the so-called \textit{normalized relative discriminative score} (NRDS), which introduces the idea of relative comparison, rather than providing absolute estimates like existing metrics.