Tracking Finite-Time Lyapunov Exponents to Robustify Neural ODEs

TL;DR

This paper explores the use of finite-time Lyapunov exponents (FTLEs) to analyze and improve the robustness of neural ODEs, providing interpretability tools and a new regularization method that enhances adversarial resilience.

Contribution

It introduces FTLE-based analysis for neural ODEs, linking Lyapunov exponents to adversarial vulnerability, and proposes a novel, computationally efficient regularization technique to improve robustness.

Findings

FTLEs effectively organize input-output dynamics.

FTLE regularization improves neural ODE robustness.

Early-stage FTLE suppression reduces computational cost.

Abstract

We investigate finite-time Lyapunov exponents (FTLEs), a measure for exponential separation of input perturbations, of deep neural networks within the framework of continuous-depth neural ODEs. We demonstrate that FTLEs are powerful organizers for input-output dynamics, allowing for better interpretability and the comparison of distinct model architectures. We establish a direct connection between Lyapunov exponents and adversarial vulnerability, and propose a novel training algorithm that improves robustness by FTLE regularization. The key idea is to suppress exponents far from zero in the early stage of the input dynamics. This approach enhances robustness and reduces computational cost compared to full-interval regularization, as it avoids a full ``double'' backpropagation.