Incorporating Inductive Biases to Energy-based Generative Models

TL;DR

This paper introduces a hybrid energy-based model that combines neural network energies with exponential family statistics to better incorporate inductive biases, improving data fitting and generation.

Contribution

The work presents a novel hybrid approach integrating exponential family statistics into energy-based models to enhance their inductive bias and modeling capabilities.

Findings

Hybrid model effectively matches data statistics.

Incorporating informative statistics improves data fitting.

Generation quality benefits from the hybrid approach.

Abstract

With the advent of score-matching techniques for model training and Langevin dynamics for sample generation, energy-based models (EBMs) have gained renewed interest as generative models. Recent EBMs usually use neural networks to define their energy functions. In this work, we introduce a novel hybrid approach that combines an EBM with an exponential family model to incorporate inductive bias into data modeling. Specifically, we augment the energy term with a parameter-free statistic function to help the model capture key data statistics. Like an exponential family model, the hybrid model aims to align the distribution statistics with data statistics during model training, even when it only approximately maximizes the data likelihood. This property enables us to impose constraints on the hybrid model. Our empirical study validates the hybrid model's ability to match statistics. Furthermore, experimental results show that data fitting and generation improve when suitable informative statistics are incorporated into the hybrid model.