Bounds all around: training energy-based models with bidirectional bounds

TL;DR

This paper introduces bidirectional bounds for training energy-based models, improving stability and quality in density estimation and sample generation by linking bounds to practical training techniques.

Contribution

It proposes a novel bidirectional bounding approach for EBMs, connecting theoretical bounds to stabilization methods and developing an efficient estimator for the Jacobian determinant.

Findings

Training stability is significantly improved.

High-quality density estimation achieved.

Sample generation quality is enhanced.

Abstract

Energy-based models (EBMs) provide an elegant framework for density estimation, but they are notoriously difficult to train. Recent work has established links to generative adversarial networks, where the EBM is trained through a minimax game with a variational value function. We propose a bidirectional bound on the EBM log-likelihood, such that we maximize a lower bound and minimize an upper bound when solving the minimax game. We link one bound to a gradient penalty that stabilizes training, thereby providing grounding for best engineering practice. To evaluate the bounds we develop a new and efficient estimator of the Jacobi-determinant of the EBM generator. We demonstrate that these developments significantly stabilize training and yield high-quality density estimation and sample generation.