# Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights

## TL;DR

This paper proves that in wide neural networks with orthogonal Haar-distributed weights, the Jacobians become asymptotically free, aiding understanding of gradient behavior and training stability.

## Contribution

It establishes the almost sure asymptotic freeness of layer-wise Jacobians in deep neural networks with orthogonal weight initialization, advancing free probability theory applications.

## Key findings

- Jacobians become asymptotically free in the wide limit
- Orthogonal Haar-distributed weights lead to well-conditioned Jacobian spectra
- Results help improve understanding of gradient stability in deep networks

## Abstract

A well-conditioned Jacobian spectrum has a vital role in preventing exploding or vanishing gradients and speeding up learning of deep neural networks. Free probability theory helps us to understand and handle the Jacobian spectrum. We rigorously show almost sure asymptotic freeness of layer-wise Jacobians of deep neural networks as the wide limit. In particular, we treat the case that weights are initialized as Haar distributed orthogonal matrices.

---
Source: https://tomesphere.com/paper/1908.03901