Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlaps

TL;DR

This paper introduces a new algorithm for efficiently learning one-hidden-layer convolutional neural networks with overlapping patches, applicable to common computer vision structures, combining isotonic regression and landscape analysis techniques.

Contribution

It presents a novel algorithm that handles overlapping patches in CNNs, advancing provable learning methods for neural networks with complex structures.

Findings

Algorithm effectively learns CNNs with overlaps

Applicable to general patch structures in vision tasks

Provides theoretical insights into non-convex optimization landscapes

Abstract

We propose a new algorithm to learn a one-hidden-layer convolutional neural network where both the convolutional weights and the outputs weights are parameters to be learned. Our algorithm works for a general class of (potentially overlapping) patches, including commonly used structures for computer vision tasks. Our algorithm draws ideas from (1) isotonic regression for learning neural networks and (2) landscape analysis of non-convex matrix factorization problems. We believe these findings may inspire further development in designing provable algorithms for learning neural networks and other complex models.