Simple2Complex: Global Optimization by Gradient Descent

TL;DR

The paper introduces Simple2Complex, a progressive training method for deep neural networks that incrementally adds layers to improve global optimization and reduce local minima trapping, demonstrated on CIFAR-10.

Contribution

It proposes a novel layer-wise growth training approach for deep networks, enhancing global optimization over traditional end-to-end methods.

Findings

Outperforms end-to-end training on CIFAR-10

Reduces likelihood of local minima trapping

Demonstrates effective layer-wise growth training

Abstract

A method named simple2complex for modeling and training deep neural networks is proposed. Simple2complex train deep neural networks by smoothly adding more and more layers to the shallow networks, as the learning procedure going on, the network is just like growing. Compared with learning by end2end, simple2complex is with less possibility trapping into local minimal, namely, owning ability for global optimization. Cifar10 is used for verifying the superiority of simple2complex.