# Layer Dynamics of Linearised Neural Nets

## TL;DR

This paper investigates the learning dynamics of multi-layer linear neural networks, revealing how layers grow at similar rates and identifying distinct learning phases, providing foundational insights into neural network behavior.

## Contribution

It extends the understanding of layer dynamics in linear neural nets and explores how nonlinearity affects these dynamics, offering a first-principles perspective.

## Key findings

- Multiple layers grow at similar rates during learning
- Distinct phases of learning with different layer growth patterns
- Nonlinearity breaks the symmetry observed in linear networks

## Abstract

Despite the phenomenal success of deep learning in recent years, there remains a gap in understanding the fundamental mechanics of neural nets. More research is focussed on handcrafting complex and larger networks, and the design decisions are often ad-hoc and based on intuition. Some recent research has aimed to demystify the learning dynamics in neural nets by attempting to build a theory from first principles, such as characterising the non-linear dynamics of specialised \textit{linear} deep neural nets (such as orthogonal networks). In this work, we expand and derive properties of learning dynamics respected by general multi-layer linear neural nets. Although an over-parameterisation of a single layer linear network, linear multi-layer neural nets offer interesting insights that explain how learning dynamics proceed in small pockets of the data space. We show in particular that multiple layers in linear nets grow at approximately the same rate, and there are distinct phases of learning with markedly different layer growth. We then apply a linearisation process to a general RelU neural net and show how nonlinearity breaks down the growth symmetry observed in liner neural nets. Overall, our work can be viewed as an initial step in building a theory for understanding the effect of layer design on the learning dynamics from first principles.