Directional Convergence Analysis under Spherically Symmetric Distribution

TL;DR

This paper proves directional convergence guarantees for simple neural networks trained on spherically symmetric data, highlighting dynamics from initialization without relying on initial loss or perfect classification.

Contribution

It provides the first directional convergence analysis with exact rates for two-layer non-linear and linear networks under spherically symmetric data distributions, starting from initialization.

Findings

Directional convergence guarantees with exact rates.

Analysis applies to two-layer non-linear and linear networks.

Results do not depend on initial loss or perfect classification.

Abstract

We consider the fundamental problem of learning linear predictors (i.e., separable datasets with zero margin) using neural networks with gradient flow or gradient descent. Under the assumption of spherically symmetric data distribution, we show directional convergence guarantees with exact convergence rate for two-layer non-linear networks with only two hidden nodes, and (deep) linear networks. Moreover, our discovery is built on dynamic from the initialization without both initial loss and perfect classification constraint in contrast to previous works. We also point out and study the challenges in further strengthening and generalizing our results.