Frobenius-Type Norms and Inner Products of Matrices and Linear Maps with Applications to Neural Network Training

TL;DR

This paper broadens the understanding of Frobenius norms and inner products for matrices and linear maps, revealing a family of norms that can be used to improve neural network training through preconditioning.

Contribution

It introduces a generalization of the Frobenius norm and inner product, showing their dependence on domain and co-domain inner products, enabling new preconditioning techniques for neural networks.

Findings

Frobenius-type norms depend on domain and co-domain inner products.

The classical Frobenius norm is a special case within a broader family.

These generalized norms can be used to enhance neural network training.

Abstract

The Frobenius norm is a frequent choice of norm for matrices. In particular, the underlying Frobenius inner product is typically used to evaluate the gradient of an objective with respect to matrix variable, such as those occuring in the training of neural networks. We provide a broader view on the Frobenius norm and inner product for linear maps or matrices, and establish their dependence on inner products in the domain and co-domain spaces. This shows that the classical Frobenius norm is merely one special element of a family of more general Frobenius-type norms. The significant extra freedom furnished by this realization can be used, among other things, to precondition neural network training.