# Convergence Results for Neural Networks via Electrodynamics

## TL;DR

This paper establishes a novel connection between neural network training convergence and an electrodynamics problem, proving that gradient descent can learn entire networks by analyzing equilibrium states akin to electrostatic forces.

## Contribution

It introduces a new analogy between neural network training and electrodynamics, demonstrating convergence properties and the possibility of learning entire networks incrementally.

## Key findings

- Gradient descent convergence is equivalent to an electrostatic equilibrium problem.
- Existence of an activation function enabling learning of at least one hidden node.
- Gradient descent can learn the entire network one node at a time.

## Abstract

We study whether a depth two neural network can learn another depth two network using gradient descent. Assuming a linear output node, we show that the question of whether gradient descent converges to the target function is equivalent to the following question in electrodynamics: Given $k$ fixed protons in $\mathbb{R}^d,$ and $k$ electrons, each moving due to the attractive force from the protons and repulsive force from the remaining electrons, whether at equilibrium all the electrons will be matched up with the protons, up to a permutation. Under the standard electrical force, this follows from the classic Earnshaw's theorem. In our setting, the force is determined by the activation function and the input distribution. Building on this equivalence, we prove the existence of an activation function such that gradient descent learns at least one of the hidden nodes in the target network. Iterating, we show that gradient descent can be used to learn the entire network one node at a time.