Simulation-Free Training of Neural ODEs on Paired Data

TL;DR

This paper introduces a simulation-free training method for Neural ODEs using flow matching in learned embedding spaces, enabling efficient learning of deterministic mappings with fewer function evaluations.

Contribution

It extends flow matching to embedding spaces for stable, simulation-free training of Neural ODEs on paired data, improving efficiency and stability.

Findings

Outperforms existing NODE methods on regression and classification tasks.

Requires significantly fewer function evaluations.

Ensures valid flows by embedding data pairs in learned spaces.

Abstract

In this work, we investigate a method for simulation-free training of Neural Ordinary Differential Equations (NODEs) for learning deterministic mappings between paired data. Despite the analogy of NODEs as continuous-depth residual networks, their application in typical supervised learning tasks has not been popular, mainly due to the large number of function evaluations required by ODE solvers and numerical instability in gradient estimation. To alleviate this problem, we employ the flow matching framework for simulation-free training of NODEs, which directly regresses the parameterized dynamics function to a predefined target velocity field. Contrary to generative tasks, however, we show that applying flow matching directly between paired data can often lead to an ill-defined flow that breaks the coupling of the data pairs (e.g., due to crossing trajectories). We propose a simple extension that applies flow matching in the embedding space of data pairs, where the embeddings are learned jointly with the dynamic function to ensure the validity of the flow which is also easier to learn. We demonstrate the effectiveness of our method on both regression and classification tasks, where our method outperforms existing NODEs with a significantly lower number of function evaluations. The code is available at https://github.com/seminkim/simulation-free-node.