Latent Time Neural Ordinary Differential Equations

TL;DR

This paper introduces LT-NODE and ALT-NODE, novel Bayesian approaches to neural ODEs that model uncertainty over the end-time parameter, improving robustness and enabling better model selection.

Contribution

The paper proposes a new Bayesian framework for neural ODEs that models uncertainty over the end-time, including a novel adaptive version for individual data points.

Findings

Effective uncertainty modeling demonstrated on synthetic data.

Improved robustness and model selection in image classification tasks.

Efficient prediction using a single forward pass with posterior samples.

Abstract

Neural ordinary differential equations (NODE) have been proposed as a continuous depth generalization to popular deep learning models such as Residual networks (ResNets). They provide parameter efficiency and automate the model selection process in deep learning models to some extent. However, they lack the much-required uncertainty modelling and robustness capabilities which are crucial for their use in several real-world applications such as autonomous driving and healthcare. We propose a novel and unique approach to model uncertainty in NODE by considering a distribution over the end-time $T$ of the ODE solver. The proposed approach, latent time NODE (LT-NODE), treats $T$ as a latent variable and apply Bayesian learning to obtain a posterior distribution over $T$ from the data. In particular, we use variational inference to learn an approximate posterior and the model parameters. Prediction is done by considering the NODE representations from different samples of the posterior and can be done efficiently using a single forward pass. As $T$ implicitly defines the depth of a NODE, posterior distribution over $T$ would also help in model selection in NODE. We also propose, adaptive latent time NODE (ALT-NODE), which allow each data point to have a distinct posterior distribution over end-times. ALT-NODE uses amortized variational inference to learn an approximate posterior using inference networks. We demonstrate the effectiveness of the proposed approaches in modelling uncertainty and robustness through experiments on synthetic and several real-world image classification data.