Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

TL;DR

This paper presents a simple, efficient parametrization for orthogonal and unitary constraints in neural networks using Lie group theory, enabling easier optimization and improved RNN performance.

Contribution

It introduces a Lie group-based exponential map parametrization that simplifies optimization under orthogonal constraints in neural networks.

Findings

Faster and more stable convergence in RNN training.

Efficient implementation with negligible runtime overhead.

Improved robustness in orthogonal constrained optimization.

Abstract

We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.