Learning Non-overlapping Convolutional Neural Networks with Multiple Kernels

TL;DR

This paper proves that non-overlapping CNNs with multiple kernels can be reliably trained to global optimality using gradient descent with tensor initialization, under certain conditions on data distribution and sample size.

Contribution

It introduces the first theoretical guarantees for parameter recovery in multi-kernel CNNs with polynomial sample and computational complexity.

Findings

Squared loss is locally strongly convex near global optima for common activations.

Tensor methods can initialize parameters within the convex basin, enabling convergence.

Gradient descent with tensor initialization guarantees global optimality under specified conditions.

Abstract

In this paper, we consider parameter recovery for non-overlapping convolutional neural networks (CNNs) with multiple kernels. We show that when the inputs follow Gaussian distribution and the sample size is sufficiently large, the squared loss of such CNNs is $\mathit{~locally~strongly~convex}$ in a basin of attraction near the global optima for most popular activation functions, like ReLU, Leaky ReLU, Squared ReLU, Sigmoid and Tanh. The required sample complexity is proportional to the dimension of the input and polynomial in the number of kernels and a condition number of the parameters. We also show that tensor methods are able to initialize the parameters to the local strong convex region. Hence, for most smooth activations, gradient descent following tensor initialization is guaranteed to converge to the global optimal with time that is linear in input dimension, logarithmic in precision and polynomial in other factors. To the best of our knowledge, this is the first work that provides recovery guarantees for CNNs with multiple kernels under polynomial sample and computational complexities.