Finite-time Lyapunov exponents of deep neural networks

TL;DR

This paper explores the analogy between deep neural networks and dynamical systems by analyzing how small input perturbations grow or decay, revealing geometric structures in input space linked to the network's classification regions.

Contribution

It introduces the concept of finite-time Lyapunov exponents to characterize the local stability and geometry of deep neural networks' input space.

Findings

Maximal Lyapunov exponents form geometrical structures in input space.

Large positive exponent ridges divide input space into class regions.

These structures illuminate the geometry underlying neural network learning.

Abstract

We compute how small input perturbations affect the output of deep neural networks, exploring an analogy between deep networks and dynamical systems, where the growth or decay of local perturbations is characterised by finite-time Lyapunov exponents. We show that the maximal exponent forms geometrical structures in input space, akin to coherent structures in dynamical systems. Ridges of large positive exponents divide input space into different regions that the network associates with different classes. These ridges visualise the geometry that deep networks construct in input space, shedding light on the fundamental mechanisms underlying their learning capabilities.