FlowAdam: Implicit Regularization via Geometry-Aware Soft Momentum Injection

TL;DR

FlowAdam is a hybrid optimizer combining Adam with ODE-based gradient flow, providing implicit regularization and improved performance on complex, coupled optimization tasks.

Contribution

It introduces Soft Momentum Injection, a novel method blending ODE velocity with Adam's momentum to enhance training stability and regularization.

Findings

Reduces error by 10-22% on matrix/tensor recovery tasks.

Achieves 6% improvement on Jester collaborative filtering.

Outperforms tuned Lion and AdaBelief optimizers in benchmarks.

Abstract

Adaptive moment methods such as Adam use a diagonal, coordinate-wise preconditioner based on exponential moving averages of squared gradients. This diagonal scaling is coordinate-system dependent and can struggle with dense or rotated parameter couplings, including those in matrix factorization, tensor decomposition, and graph neural networks, because it treats each parameter independently. We introduce FlowAdam, a hybrid optimizer that augments Adam with continuous gradient-flow integration via an ordinary differential equation (ODE). When EMA-based statistics detect landscape difficulty, FlowAdam switches to clipped ODE integration. Our central contribution is Soft Momentum Injection, which blends ODE velocity with Adam's momentum during mode transitions. This prevents the training collapse observed with naive hybrid approaches. Across coupled optimization benchmarks, the ODE integration provides implicit regularization, reducing held-out error by 10-22% on low-rank matrix/tensor recovery and 6% on Jester (real-world collaborative filtering), also surpassing tuned Lion and AdaBelief, while matching Adam on well-conditioned workloads (CIFAR-10). MovieLens-100K confirms benefits arise specifically from coupled parameter interactions rather than bias estimation. Ablation studies show that soft injection is essential, as hard replacement reduces accuracy from 100% to 82.5%.