Lookahead Drifting Model

TL;DR

The paper introduces a lookahead drifting model that computes multiple drifting terms sequentially to improve distribution mapping, demonstrating better performance on toy examples and CIFAR10.

Contribution

It proposes a novel lookahead drifting approach that captures higher order gradient information for improved image generation performance.

Findings

Outperforms baseline on toy examples.

Achieves better results on CIFAR10.

Effectively captures higher order gradient information.

Abstract

Recently, a new paradigm named \emph{drifting model} has been proposed for mapping distributions, which achieves the SOTA image generation performance over ImageNet via one-step neural functional evaluation (NFE). The basic idea is to compute a drifting term at each training iteration and then push the output of the model towards the direction of the drifting term. In this paper, we propose a \emph{lookahead drifting model}. At each training iteration, we compute a set of drifting terms sequentially. Each drifting term is calculated by making use of previously computed ones as well as the positive samples and the output of the model. %One key step is to properly scale the drifting terms so that their magnitudes are in a comparable range. In principle, the drifting terms obtained at a later stage capture higher order gradient information towards the positive samples. At each training iteration, the model is optimized by pushing its output towards the direction of the (weighted) summation of the drifting terms. Experimental results on toy examples and CIFAR10 demonstrate the better performance of the new method than the baseline.