# On Symmetry and Initialization for Neural Networks

## TL;DR

This paper explores how symmetry and careful initialization in neural networks with one hidden layer can lead to efficient training and generalization, supported by theoretical analysis and empirical validation.

## Contribution

It demonstrates that symmetry-aware initialization improves training efficiency and guarantees in neural networks, unlike random initializations.

## Key findings

- Symmetry-aware initialization enhances convergence.
- Random initializations do not guarantee efficient learning.
- Theoretical analysis links symmetry to training success.

## Abstract

This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.