Learning Implicit Generative Models by Teaching Explicit Ones

TL;DR

This paper introduces a novel learning by teaching (LBT) approach for implicit generative models that avoids mode collapse by optimizing a KL-divergence and incorporates an auxiliary density estimator, enhancing training stability and effectiveness.

Contribution

The paper proposes a new LBT framework that combines implicit models with explicit density estimators, formulated as a bilevel optimization problem, and integrates with GANs for improved generative modeling.

Findings

LBT effectively avoids mode collapse in implicit models.

Hybrid LBT-GAN benefits from combined strengths of both methods.

Experimental results show improved training stability and quality of generated data.

Abstract

Implicit generative models are difficult to train as no explicit density functions are defined. Generative adversarial nets (GANs) present a minimax framework to train such models, which however can suffer from mode collapse due to the nature of the JS-divergence. This paper presents a learning by teaching (LBT) approach to learning implicit models, which intrinsically avoids the mode collapse problem by optimizing a KL-divergence rather than the JS-divergence in GANs. In LBT, an auxiliary density estimator is introduced to fit the implicit model's distribution while the implicit model teaches the density estimator to match the data distribution. LBT is formulated as a bilevel optimization problem, whose optimal generator matches the true data distribution. LBT can be naturally integrated with GANs to derive a hybrid LBT-GAN that enjoys complimentary benefits. Finally, we present a stochastic gradient ascent algorithm with unrolling to solve the challenging learning problems. Experimental results demonstrate the effectiveness of our method.