Deep Networks Provably Classify Data on Curves

TL;DR

This paper proves that deep neural networks can reliably classify data lying on smooth curves on the sphere, with guarantees depending on the network depth and intrinsic data properties, using NTK analysis.

Contribution

It provides the first generalization guarantee for deep networks classifying nonlinear data based solely on intrinsic geometric properties.

Findings

Deep networks with sufficient depth can classify data on curves with high probability.

The neural tangent kernel (NTK) can be approximated by a translationally invariant operator for deep networks.

Deeper networks improve the stability and invertibility of the NTK, aiding classification.

Abstract

Data with low-dimensional nonlinear structure are ubiquitous in engineering and scientific problems. We study a model problem with such structure -- a binary classification task that uses a deep fully-connected neural network to classify data drawn from two disjoint smooth curves on the unit sphere. Aside from mild regularity conditions, we place no restrictions on the configuration of the curves. We prove that when (i) the network depth is large relative to certain geometric properties that set the difficulty of the problem and (ii) the network width and number of samples is polynomial in the depth, randomly-initialized gradient descent quickly learns to correctly classify all points on the two curves with high probability. To our knowledge, this is the first generalization guarantee for deep networks with nonlinear data that depends only on intrinsic data properties. Our analysis proceeds by a reduction to dynamics in the neural tangent kernel (NTK) regime, where the network depth plays the role of a fitting resource in solving the classification problem. In particular, via fine-grained control of the decay properties of the NTK, we demonstrate that when the network is sufficiently deep, the NTK can be locally approximated by a translationally invariant operator on the manifolds and stably inverted over smooth functions, which guarantees convergence and generalization.