A Fully Rigorous Proof of the Derivation of Xavier and He's Initialization for Deep ReLU Networks

TL;DR

This paper provides a comprehensive and rigorous mathematical proof explaining how Xavier and He's initialization methods are derived for deep neural networks with ReLU activation functions.

Contribution

It offers the first fully rigorous derivation of Xavier and He's initialization techniques for ReLU networks, clarifying their theoretical foundations.

Findings

Formal proof of Xavier and He's initialization derivation

Enhanced understanding of initialization impact on ReLU networks

Theoretical validation of initialization methods

Abstract

A fully rigorous proof of the derivation of Xavier/He's initialization for ReLU nets is given.