Energy-based generator matching: A neural sampler for general state space

TL;DR

This paper introduces Energy-based generator matching (EGM), a versatile neural sampling method that trains generative models directly from energy functions without relying on data, applicable across various modalities and continuous-time processes.

Contribution

EGM extends generator matching to train arbitrary continuous-time Markov processes, enabling modality-agnostic data generation from energy functions.

Findings

Validated on discrete and multimodal tasks

Effective in high-dimensional settings

Reduces variance with importance sampling

Abstract

We propose Energy-based generator matching (EGM), a modality-agnostic approach to train generative models from energy functions in the absence of data. Extending the recently proposed generator matching, EGM enables training of arbitrary continuous-time Markov processes, e.g., diffusion, flow, and jump, and can generate data from continuous, discrete, and a mixture of two modalities. To this end, we propose estimating the generator matching loss using self-normalized importance sampling with an additional bootstrapping trick to reduce variance in the importance weight. We validate EGM on both discrete and multimodal tasks up to 100 and 20 dimensions, respectively.