On Alignment in Deep Linear Neural Networks

TL;DR

This paper investigates the concept of alignment in deep linear neural networks, exploring its role as an implicit regularizer during gradient descent, and characterizes conditions under which alignment is invariant and influences convergence.

Contribution

It extends the concept of alignment to networks with multidimensional outputs, analyzes its invariance during training, and examines the effects of layer constraints like convolutional structures.

Findings

Alignment exists as a global minimum in fully connected networks.

Alignment invariance depends on specific conditions during training.

Gradient descent converges linearly under certain learning rate conditions.

Abstract

We study the properties of alignment, a form of implicit regularization, in linear neural networks under gradient descent. We define alignment for fully connected networks with multidimensional outputs and show that it is a natural extension of alignment in networks with 1-dimensional outputs as defined by Ji and Telgarsky, 2018. While in fully connected networks, there always exists a global minimum corresponding to an aligned solution, we analyze alignment as it relates to the training process. Namely, we characterize when alignment is an invariant of training under gradient descent by providing necessary and sufficient conditions for this invariant to hold. In such settings, the dynamics of gradient descent simplify, thereby allowing us to provide an explicit learning rate under which the network converges linearly to a global minimum. We then analyze networks with layer constraints such as convolutional networks. In this setting, we prove that gradient descent is equivalent to projected gradient descent, and that alignment is impossible with sufficiently large datasets.