Tail Annealing for Heavy-Tailed Flow Matching

TL;DR

This paper introduces a simple data transformation technique called tail annealing, which enables flow models to better handle heavy-tailed data by compressing tails into a manageable range, improving performance on benchmarks.

Contribution

The authors propose a novel tail annealing method using a soft-log transform combined with a Hill diagnostic, enhancing flow matching for heavy-tailed distributions without complex architectural changes.

Findings

Log-FM outperforms specialized baselines on heavy-tailed benchmarks.

Zero severe divergences observed across extensive experiments.

Transform effectively maps Pareto tails to exponential, aiding flow modeling.

Abstract

Standard generative models struggle with heavy-tailed data: Lipschitz architectures cannot produce power-law tails from Gaussian noise, and interpolating between heavy-tailed data and Gaussians is ill-posed. We propose a simple fix: apply the soft-log transform $\phi(x) = \mathrm{sign}(x) \cdot \log(1 + |x|)$ coordinate-wise to data before training, then exponentiate samples after generation. A Hill diagnostic decides per-coordinate whether to transform, leaving light-tailed margins untouched at no added complexity. This compresses heavy tails into a range where standard flow matching succeeds, without heavy-tailed base distributions or architectural modifications. We provide theoretical intuition for why this works: the log-transform maps Pareto tails to exponentials, and the induced dynamics implement a form of tail annealing via power transformations. On a 144-configuration multivariate benchmark (3 copulas, $d$ up to 100, 4 tail indices), Log-FM dominates specialized baselines on $W_1$, CVaR$_{99}$, and extreme-quantile metrics, and is the only method with zero severe divergences across 2{,}880 runs.