A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes

TL;DR

This paper extends a Riemannian geometric framework to various neural network architectures, including non-differentiable activations, providing insights into their functioning through numerical experiments on image and thermodynamic data.

Contribution

It generalizes the geometric approach to convolutional, residual, and recursive neural networks, including non-differentiable activation functions like ReLU.

Findings

Extended geometric framework to new neural network architectures

Analyzed non-differentiable activation functions within this framework

Demonstrated effectiveness through numerical experiments on classification tasks

Abstract

Neural networks are playing a crucial role in everyday life, with the most modern generative models able to achieve impressive results. Nonetheless, their functioning is still not very clear, and several strategies have been adopted to study how and why these model reach their outputs. A common approach is to consider the data in an Euclidean settings: recent years has witnessed instead a shift from this paradigm, moving thus to more general framework, namely Riemannian Geometry. Two recent works introduced a geometric framework to study neural networks making use of singular Riemannian metrics. In this paper we extend these results to convolutional, residual and recursive neural networks, studying also the case of non-differentiable activation functions, such as ReLU. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.