One Reflection Suffice

TL;DR

This paper proposes a novel method to efficiently enforce orthogonal weight matrices in deep learning by using a single Householder reflection computed via an auxiliary neural network, reducing computational overhead.

Contribution

It introduces a theoretically grounded approach that replaces multiple reflections with a single learned reflection, improving GPU utilization in orthogonal matrix constraints.

Findings

Single reflection suffices for orthogonality

Improved GPU utilization demonstrated

Theoretical proof of reflection sufficiency

Abstract

Orthogonal weight matrices are used in many areas of deep learning. Much previous work attempt to alleviate the additional computational resources it requires to constrain weight matrices to be orthogonal. One popular approach utilizes *many* Householder reflections. The only practical drawback is that many reflections cause low GPU utilization. We mitigate this final drawback by proving that *one* reflection is sufficient, if the reflection is computed by an auxiliary neural network.