Riemannian metrics for neural networks I: feedforward networks

TL;DR

This paper introduces four mathematically principled algorithms for neural network training based on Riemannian geometry, invariant to data and network transformations, enhancing scalability and performance.

Contribution

It presents novel Riemannian-based algorithms for neural network training that are scalable and invariant under various transformations, bridging differential geometry with deep learning optimization.

Findings

Algorithms are invariant under data and network transformations

Methods improve scalability of Riemannian optimization in neural networks

Performance is consistent across different data and network representations

Abstract

We describe four algorithms for neural network training, each adapted to different scalability constraints. These algorithms are mathematically principled and invariant under a number of transformations in data and network representation, from which performance is thus independent. These algorithms are obtained from the setting of differential geometry, and are based on either the natural gradient using the Fisher information matrix, or on Hessian methods, scaled down in a specific way to allow for scalability while keeping some of their key mathematical properties.