Stability of Deep Neural Networks via discrete rough paths

TL;DR

This paper applies rough path theory to analyze the stability of deep residual neural networks, providing bounds that remain valid even when weights are highly irregular, such as Brownian motion-like functions.

Contribution

It introduces a novel stability analysis framework for neural networks using discrete rough paths, accommodating very rough weights beyond classical smooth assumptions.

Findings

Stability bounds depend on the total p-variation of weights for p in [1,3]

Estimates remain bounded for weights like Brownian motions

Residual networks can be viewed as solutions to rough difference equations

Abstract

Using rough path techniques, we provide a priori estimates for the output of Deep Residual Neural Networks in terms of both the input data and the (trained) network weights. As trained network weights are typically very rough when seen as functions of the layer, we propose to derive stability bounds in terms of the total $p$-variation of trained weights for any $p\in[1,3]$. Unlike the $C^1$-theory underlying the neural ODE literature, our estimates remain bounded even in the limiting case of weights behaving like Brownian motions, as suggested in [arXiv:2105.12245]. Mathematically, we interpret residual neural network as solutions to (rough) difference equations, and analyse them based on recent results of discrete time signatures and rough path theory.