Analytical Correction for Subsampling Bias in Drifting Models

TL;DR

This paper introduces Analytical Bias Correction (ABC), a simple and effective method to reduce subsampling bias in drifting models, improving accuracy and training speed especially with small sample sizes.

Contribution

The paper proposes a closed-form bias correction method that reduces bias from $O(1/n)$ to $O(1/n^2)$ in drifting models, with minimal computational overhead.

Findings

ABC effectively reduces bias in experiments.

ABC improves FID scores on CIFAR-10.

Bias correction accelerates training with small batch sizes.

Abstract

Drifting models are capable one-step generative models trained to follow a drifting field. The field combines attractive and repulsive softmax-weighted centroids over the data and current-generator distributions. In practice, only a minibatch of $n$ samples from each distribution is available, and each centroid is approximated by an empirical estimate. In this paper, we begin by showing that the minibatch centroid is in general a biased estimator of the target centroid, with a pointwise $O(1/n)$ bias arising from softmax self-normalization. Correcting this bias requires the expectation over the full distribution, which is intractable. We instead approximate the leading bias term from in-batch statistics and propose Analytical Bias Correction (ABC), a closed-form plug-in adjustment. We prove that ABC reduces the bias from $O(1/n)$ to $O(1/n^2)$, introduces no first-order increase in total variance, and preserves convex-hull containment of the corrected centroid. In practice, ABC requires only two additional lines of code and has negligible wall-time overhead under compiled execution. Toy experiments confirm the theoretical $O(1/n)$ and $O(1/n^2)$ scaling. On CIFAR-10, ABC reduces FID and trains faster, with the largest gains at small $n$, where the bias is most significant.