Deep neural networks from the perspective of ergodic theory

TL;DR

This paper explores deep neural networks through the lens of ergodic theory, proposing that viewing networks as dynamical systems can clarify design heuristics and provide a more scientific understanding.

Contribution

It introduces a novel perspective by applying ergodic theory to neural networks, offering heuristic explanations for network design principles.

Findings

Networks can be modeled as dynamical systems with layers as time steps

Ergodic theory provides insights into the behavior of deep networks

Design heuristics can be understood as emergent properties of ergodic dynamics

Abstract

The design of deep neural networks remains somewhat of an art rather than precise science. By tentatively adopting ergodic theory considerations on top of viewing the network as the time evolution of a dynamical system, with each layer corresponding to a temporal instance, we show that some rules of thumb, which might otherwise appear mysterious, can be attributed heuristics.