Neural Polytopes

TL;DR

This paper demonstrates that simple neural networks with ReLU activation generate polytopes approximating spheres, with their structure influenced by network architecture, and introduces neural polytopes as a smooth generalization linking geometry and machine learning.

Contribution

It reveals how neural networks produce polytopes as geometric approximations and introduces neural polytopes as a novel concept bridging discrete geometry and neural network representations.

Findings

Neural networks with ReLU generate polytopes approximating spheres.

Network architecture influences the type of generated polytopes.

Neural polytopes generalize polytopes to smooth geometric structures.

Abstract

We find that simple neural networks with ReLU activation generate polytopes as an approximation of a unit sphere in various dimensions. The species of polytopes are regulated by the network architecture, such as the number of units and layers. For a variety of activation functions, generalization of polytopes is obtained, which we call neural polytopes. They are a smooth analogue of polytopes, exhibiting geometric duality. This finding initiates research of generative discrete geometry to approximate surfaces by machine learning.