Sparsity in Continuous-Depth Neural Networks

TL;DR

This paper investigates how different types of sparsity in neural ordinary differential equations (NODEs) affect their ability to generalize and identify underlying dynamical laws, proposing new regularization methods and testing on real-world datasets.

Contribution

It introduces a regularization technique for sparsifying input-output connections in NODEs and provides a comprehensive empirical evaluation on real-world datasets for forecasting and dynamics identification.

Findings

Weight sparsity enhances generalization under noise and irregular sampling.

Feature sparsity aids in recovering true sparse dynamics.

Spurious feature dependencies can still be learned despite sparsity.

Abstract

Neural Ordinary Differential Equations (NODEs) have proven successful in learning dynamical systems in terms of accurately recovering the observed trajectories. While different types of sparsity have been proposed to improve robustness, the generalization properties of NODEs for dynamical systems beyond the observed data are underexplored. We systematically study the influence of weight and feature sparsity on forecasting as well as on identifying the underlying dynamical laws. Besides assessing existing methods, we propose a regularization technique to sparsify "input-output connections" and extract relevant features during training. Moreover, we curate real-world datasets consisting of human motion capture and human hematopoiesis single-cell RNA-seq data to realistically analyze different levels of out-of-distribution (OOD) generalization in forecasting and dynamics identification respectively. Our extensive empirical evaluation on these challenging benchmarks suggests that weight sparsity improves generalization in the presence of noise or irregular sampling. However, it does not prevent learning spurious feature dependencies in the inferred dynamics, rendering them impractical for predictions under interventions, or for inferring the true underlying dynamics. Instead, feature sparsity can indeed help with recovering sparse ground-truth dynamics compared to unregularized NODEs.