Continuum Dropout for Neural Differential Equations

TL;DR

This paper introduces Continuum Dropout, a novel regularization method for Neural Differential Equations that improves generalization, prevents overfitting, and provides uncertainty quantification through a stochastic continuous-time dropout process.

Contribution

It proposes a theoretically grounded Continuum Dropout technique based on renewal processes, specifically designed for NDEs, enhancing their robustness and uncertainty estimation capabilities.

Findings

Outperforms existing regularization methods on time series tasks

Achieves better calibration of probability estimates

Enhances generalization and uncertainty quantification in NDEs

Abstract

Neural Differential Equations (NDEs) excel at modeling continuous-time dynamics, effectively handling challenges such as irregular observations, missing values, and noise. Despite their advantages, NDEs face a fundamental challenge in adopting dropout, a cornerstone of deep learning regularization, making them susceptible to overfitting. To address this research gap, we introduce Continuum Dropout, a universally applicable regularization technique for NDEs built upon the theory of alternating renewal processes. Continuum Dropout formulates the on-off mechanism of dropout as a stochastic process that alternates between active (evolution) and inactive (paused) states in continuous time. This provides a principled approach to prevent overfitting and enhance the generalization capabilities of NDEs. Moreover, Continuum Dropout offers a structured framework to quantify predictive uncertainty via Monte Carlo sampling at test time. Through extensive experiments, we demonstrate that Continuum Dropout outperforms existing regularization methods for NDEs, achieving superior performance on various time series and image classification tasks. It also yields better-calibrated and more trustworthy probability estimates, highlighting its effectiveness for uncertainty-aware modeling.