Tracking High-order Evolutions via Cascading Low-rank Fitting

TL;DR

This paper introduces cascading low-rank fitting, a method to efficiently model high-order derivatives in diffusion models, reducing computational complexity while maintaining theoretical guarantees on rank behavior.

Contribution

It proposes a novel low-rank approximation approach for high-order derivatives in diffusion models, with theoretical analysis and an efficient algorithm.

Findings

High-order derivatives can be approximated with shared base functions and low-rank components.

The rank of derivatives is non-increasing under certain initial conditions.

The method allows designing derivative rank sequences with arbitrary permutations.

Abstract

Diffusion models have become the de facto standard for modern visual generation, including well-established frameworks such as latent diffusion and flow matching. Recently, modeling high-order dynamics has emerged as a promising frontier in generative modeling. Rather than only learning the first-order velocity field that transports random noise to a target data distribution, these approaches simultaneously learn higher-order derivatives, such as acceleration and jerk, yielding a diverse family of higher-order diffusion variants. To represent higher-order derivatives, naive approaches instantiate separate neural networks for each order, which scales the parameter space linearly with the derivative order. To overcome this computational bottleneck, we introduce cascading low-rank fitting, an ordinary differential equation inspired method that approximates successive derivatives by applying a shared base function augmented with sequentially accumulated low-rank components. Theoretically, we analyze the rank dynamics of these successive matrix differences. We prove that if the initial difference is linearly decomposable, the generic ranks of high-order derivatives are guaranteed to be monotonically non-increasing. Conversely, we demonstrate that without this structural assumption, the General Leibniz Rule allows ranks to strictly increase. Furthermore, we establish that under specific conditions, the sequence of derivative ranks can be designed to form any arbitrary permutation. Finally, we present a straightforward algorithm to efficiently compute the proposed cascading low-rank fitting.