EBM Life Cycle: MCMC Strategies for Synthesis, Defense, and Density Modeling

TL;DR

This paper introduces MCMC initialization strategies tailored to different sampling lengths in Energy-Based Models, enabling improved image generation, adversarial defense, and density modeling with state-of-the-art results.

Contribution

It proposes three novel MCMC initialization methods that enhance EBM training across various sampling trajectories without altering the core ML objective.

Findings

Achieved state-of-the-art FID scores on CIFAR-10 and ImageNet.

First EBM-based adversarial defense on ImageNet.

Scalable techniques for learning valid image densities.

Abstract

This work presents strategies to learn an Energy-Based Model (EBM) according to the desired length of its MCMC sampling trajectories. MCMC trajectories of different lengths correspond to models with different purposes. Our experiments cover three different trajectory magnitudes and learning outcomes: 1) shortrun sampling for image generation; 2) midrun sampling for classifier-agnostic adversarial defense; and 3) longrun sampling for principled modeling of image probability densities. To achieve these outcomes, we introduce three novel methods of MCMC initialization for negative samples used in Maximum Likelihood (ML) learning. With standard network architectures and an unaltered ML objective, our MCMC initialization methods alone enable significant performance gains across the three applications that we investigate. Our results include state-of-the-art FID scores for unnormalized image densities on the CIFAR-10 and ImageNet datasets; state-of-the-art adversarial defense on CIFAR-10 among purification methods and the first EBM defense on ImageNet; and scalable techniques for learning valid probability densities. Code for this project can be found at https://github.com/point0bar1/ebm-life-cycle.