Using Intermediate Forward Iterates for Intermediate Generator Optimization

TL;DR

This paper introduces Intermediate Generator Optimization (IGO), a novel backpropagation-based method that leverages intermediate iterates in score-based models to improve generative tasks with minimal extra computation.

Contribution

The paper proposes IGO, a new technique that utilizes intermediate process iterates in score-based models, compatible with standard autoencoders, enhancing generative modeling without significant computational costs.

Findings

IGO enables ensemble generation at various time points using first-order methods.

Application of IGO improves image extrapolation and point cloud denoising.

Sample complexity analysis shows effectiveness in downstream tasks.

Abstract

Score-based models have recently been introduced as a richer framework to model distributions in high dimensions and are generally more suitable for generative tasks. In score-based models, a generative task is formulated using a parametric model (such as a neural network) to directly learn the gradient of such high dimensional distributions, instead of the density functions themselves, as is done traditionally. From the mathematical point of view, such gradient information can be utilized in reverse by stochastic sampling to generate diverse samples. However, from a computational perspective, existing score-based models can be efficiently trained only if the forward or the corruption process can be computed in closed form. By using the relationship between the process and layers in a feed-forward network, we derive a backpropagation-based procedure which we call Intermediate Generator Optimization to utilize intermediate iterates of the process with negligible computational overhead. The main advantage of IGO is that it can be incorporated into any standard autoencoder pipeline for the generative task. We analyze the sample complexity properties of IGO to solve downstream tasks like Generative PCA. We show applications of the IGO on two dense predictive tasks viz., image extrapolation, and point cloud denoising. Our experiments indicate that obtaining an ensemble of generators for various time points is possible using first-order methods.