From Euler to Dormand-Prince: ODE Solvers for Flow Matching Generative Models

TL;DR

This paper benchmarks classical ODE solvers for flow matching generative models, showing that higher-order methods like RK4 can achieve comparable quality with fewer function evaluations than simpler methods.

Contribution

Derives and implements four classical ODE solvers from first principles for flow matching models, providing systematic efficiency benchmarks and empirical insights.

Findings

RK4 at 80 evaluations matches Euler at 200 evaluations in sample quality.

Jacobian eigenvalues stiffen near t=1, influencing solver step distribution.

Solver choice impacts quality more for undertrained or smaller models.

Abstract

Sampling from Flow Matching generative models requires solving an ordinary differential equation (ODE) whose computational cost is dominated by neural network forward passes. We derive four classical ODE solvers -- Euler, Explicit Midpoint, Classical Runge-Kutta (RK4), and Dormand-Prince 5(4) -- from first principles via Taylor expansion, implement them from scratch in PyTorch, and systematically benchmark their efficiency on Conditional Flow Matching tasks ranging from 2D toy distributions to MNIST digits. On the quantitative side, we use sliced Wasserstein distance to construct NFE-quality Pareto frontiers,finding that RK4 at 80 function evaluations achieves sample quality comparable to Euler at 200. Beyond reproducing known convergence rates, we report two empirical observations: (1) the Jacobian eigenvalue spectrum of the learned velocity field stiffens sharply near t=1, explaining why the adaptive Dormand-Prince solver automatically concentrates its step budget at the end of the trajectory; (2) the quality gap between low-order and high-order solvers widens for undertrained and smaller models, indicating that solver choice matters most when the model is imperfect. Code and all experiment scripts are publicly available.