Convexifying Sparse Interpolation with Infinitely Wide Neural Networks: An Atomic Norm Approach

TL;DR

This paper introduces a convex framework for exact data interpolation using sparse, infinitely wide neural networks with leaky ReLU activations, leveraging atomic norms to characterize the convex hulls of network parameters.

Contribution

It derives convex formulations for sparse neural network interpolation problems using atomic norms, extending to binary classification, and compares these with gradient descent training.

Findings

Convex formulations accurately interpolate data with sparse networks.

The atomic norm approach provides a new convex characterization of neural network interpolation.

Experimental results show competitive performance with traditional training methods.

Abstract

This work examines the problem of exact data interpolation via sparse (neuron count), infinitely wide, single hidden layer neural networks with leaky rectified linear unit activations. Using the atomic norm framework of [Chandrasekaran et al., 2012], we derive simple characterizations of the convex hulls of the corresponding atomic sets for this problem under several different constraints on the weights and biases of the network, thus obtaining equivalent convex formulations for these problems. A modest extension of our proposed framework to a binary classification problem is also presented. We explore the efficacy of the resulting formulations experimentally, and compare with networks trained via gradient descent.