Understanding symmetries in deep networks

TL;DR

This paper investigates complex symmetries in deep networks' weight space and proposes a unit-norm constraint to improve training stability and test performance, demonstrated on MNIST.

Contribution

It introduces a method to constrain network filters on the unit-norm manifold to address weight space symmetries in deep networks.

Findings

Unit-norm constraints improve test accuracy.

Method enhances training stability over standard batch normalization.

Empirical results on MNIST show performance gains.

Abstract

Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show that a commonly used deep network, which uses convolution, batch normalization, reLU, max-pooling, and sub-sampling pipeline, possess more complex forms of symmetry arising from scaling-based reparameterization of the network weights. We propose to tackle the issue of the weight space symmetry by constraining the filters to lie on the unit-norm manifold. Consequently, training the network boils down to using stochastic gradient descent updates on the unit-norm manifold. Our empirical evidence based on the MNIST dataset shows that the proposed updates improve the test performance beyond what is achieved with batch normalization and without sacrificing the computational efficiency of the weight updates.