Expressivity of Neural Networks via Chaotic Itineraries beyond Sharkovsky's Theorem

TL;DR

This paper explores how chaotic dynamical behaviors in neural networks can significantly enhance their expressivity, surpassing traditional periodicity-based limits, and identifies a phase transition linked to chaotic regimes affecting complexity measures.

Contribution

It demonstrates that chaotic itineraries lead to exponential expressivity gains in neural networks, improving upon prior periodicity-based bounds and revealing a phase transition in complexity.

Findings

Chaotic itineraries yield stronger exponential tradeoffs.

Bounds tighten with increasing period and are nearly optimal.

A phase transition to chaos coincides with shifts in complexity measures.

Abstract

Given a target function $f$, how large must a neural network be in order to approximate $f$? Recent works examine this basic question on neural network \textit{expressivity} from the lens of dynamical systems and provide novel ``depth-vs-width'' tradeoffs for a large family of functions $f$. They suggest that such tradeoffs are governed by the existence of \textit{periodic} points or \emph{cycles} in $f$. Our work, by further deploying dynamical systems concepts, illuminates a more subtle connection between periodicity and expressivity: we prove that periodic points alone lead to suboptimal depth-width tradeoffs and we improve upon them by demonstrating that certain ``chaotic itineraries'' give stronger exponential tradeoffs, even in regimes where previous analyses only imply polynomial gaps. Contrary to prior works, our bounds are nearly-optimal, tighten as the period increases, and handle strong notions of inapproximability (e.g., constant $L_1$ error). More broadly, we identify a phase transition to the \textit{chaotic regime} that exactly coincides with an abrupt shift in other notions of function complexity, including VC-dimension and topological entropy.