Exploring bidirectional bounds for minimax-training of Energy-based models

TL;DR

This paper introduces a novel approach to training Energy-based models using bidirectional bounds, which improves stability and quality in density estimation and sample generation.

Contribution

It proposes the use of bidirectional bounds for EBM training, including new bounds based on Jacobian singular values, mutual information, and diffusion processes.

Findings

Bidirectional bounds stabilize EBM training.

Enhanced density estimation quality.

Improved sample generation results.

Abstract

Energy-based models (EBMs) estimate unnormalized densities in an elegant framework, but they are generally difficult to train. Recent work has linked EBMs to generative adversarial networks, by noting that they can be trained through a minimax game using a variational lower bound. To avoid the instabilities caused by minimizing a lower bound, we propose to instead work with bidirectional bounds, meaning that we maximize a lower bound and minimize an upper bound when training the EBM. We investigate four different bounds on the log-likelihood derived from different perspectives. We derive lower bounds based on the singular values of the generator Jacobian and on mutual information. To upper bound the negative log-likelihood, we consider a gradient penalty-like bound, as well as one based on diffusion processes. In all cases, we provide algorithms for evaluating the bounds. We compare the different bounds to investigate, the pros and cons of the different approaches. Finally, we demonstrate that the use of bidirectional bounds stabilizes EBM training and yields high-quality density estimation and sample generation.