Learning Latent Space Hierarchical EBM Diffusion Models

TL;DR

This paper introduces a novel approach combining diffusion probabilistic schemes with energy-based models to improve learning and sampling in hierarchical latent space models, addressing prior limitations and enhancing performance.

Contribution

It proposes using diffusion schemes to effectively learn and sample from multi-modal EBM priors in hierarchical latent space models, overcoming prior sampling challenges.

Findings

Superior performance on challenging tasks

Effective mitigation of EBM sampling issues

Enhanced expressivity of hierarchical models

Abstract

This work studies the learning problem of the energy-based prior model and the multi-layer generator model. The multi-layer generator model, which contains multiple layers of latent variables organized in a top-down hierarchical structure, typically assumes the Gaussian prior model. Such a prior model can be limited in modelling expressivity, which results in a gap between the generator posterior and the prior model, known as the prior hole problem. Recent works have explored learning the energy-based (EBM) prior model as a second-stage, complementary model to bridge the gap. However, the EBM defined on a multi-layer latent space can be highly multi-modal, which makes sampling from such marginal EBM prior challenging in practice, resulting in ineffectively learned EBM. To tackle the challenge, we propose to leverage the diffusion probabilistic scheme to mitigate the burden of EBM sampling and thus facilitate EBM learning. Our extensive experiments demonstrate a superior performance of our diffusion-learned EBM prior on various challenging tasks.