Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold

TL;DR

This paper introduces a novel method for training time-varying generative models by projecting their evolution onto an exponential family manifold and using natural gradient descent, enabling efficient approximation and particle-based updates.

Contribution

It establishes a connection between generative model training and exponential family probabilistic models, proposing a natural gradient-based algorithm with particle implementations.

Findings

Effective training of time-varying generative models demonstrated

Efficient natural gradient approximation without MCMC

Particle algorithms with closed-form updates validated

Abstract

Optimising probabilistic models is a well-studied field in statistics. However, its connection with the training of generative models remains largely under-explored. In this paper, we show that the evolution of time-varying generative models can be projected onto an exponential family manifold, naturally creating a link between the parameters of a generative model and those of a probabilistic model. We then train the generative model by moving its projection on the manifold according to the natural gradient descent scheme. This approach also allows us to efficiently approximate the natural gradient of the KL divergence without relying on MCMC for intractable models. Furthermore, we propose particle versions of the algorithm, which feature closed-form update rules for any parametric model within the exponential family. Through toy and real-world experiments, we validate the effectiveness of the proposed algorithms. The code of the proposed algorithms can be found at https://github.com/anewgithubname/iNGD.