On Batch Orthogonalization Layers

TL;DR

This paper explores orthonormalization layers as an alternative to batch normalization in neural networks, comparing their performance with BN using various metrics.

Contribution

It introduces and evaluates orthonormalization layers as a novel alternative to batch normalization in neural networks.

Findings

Orthonormalization layers can serve as effective alternatives to BN.

Performance varies depending on the specific orthogonalization method used.

Some orthogonalization methods achieve comparable results to BN.

Abstract

Batch normalization has become ubiquitous in many state-of-the-art nets. It accelerates training and yields good performance results. However, there are various other alternatives to normalization, e.g. orthonormalization. The objective of this paper is to explore the possible alternatives to channel normalization with orthonormalization layers. The performance of the algorithms are compared together with BN with prescribed performance measures.