Tropical Geometry of Deep Neural Networks

TL;DR

This paper reveals a novel connection between deep neural networks with ReLU activation and tropical geometry, providing new geometric insights into their structure and expressiveness.

Contribution

It establishes the first formal link between neural networks and tropical geometry, characterizing neural network features using tropical rational maps and geometric objects.

Findings

Feedforward ReLU networks with one hidden layer relate to zonotopes.

Decision boundaries correspond to tropical hypersurfaces.

Deeper networks are exponentially more expressive than shallow ones.

Abstract

We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden layer can be characterized by zonotopes, which serve as building blocks for deeper networks; we relate decision boundaries of such neural networks to tropical hypersurfaces, a major object of study in tropical geometry; and we prove that linear regions of such neural networks correspond to vertices of polytopes associated with tropical rational functions. An insight from our tropical formulation is that a deeper network is exponentially more expressive than a shallow network.