Progressive Learning for Systematic Design of Large Neural Networks

TL;DR

This paper introduces a progressive, systematic approach for designing large neural networks by incrementally increasing network size and optimizing each layer with convex methods, reducing manual tuning and improving generalization.

Contribution

It proposes a novel progressive design algorithm leveraging the property of certain nonlinear functions, simplifying network construction and parameter regularization.

Findings

Networks designed with the method show good generalization.

Regularization reduces manual tuning effort.

Random weights help decrease learnable parameters.

Abstract

We develop an algorithm for systematic design of a large artificial neural network using a progression property. We find that some non-linear functions, such as the rectifier linear unit and its derivatives, hold the property. The systematic design addresses the choice of network size and regularization of parameters. The number of nodes and layers in network increases in progression with the objective of consistently reducing an appropriate cost. Each layer is optimized at a time, where appropriate parameters are learned using convex optimization. Regularization parameters for convex optimization do not need a significant manual effort for tuning. We also use random instances for some weight matrices, and that helps to reduce the number of parameters we learn. The developed network is expected to show good generalization power due to appropriate regularization and use of random weights in the layers. This expectation is verified by extensive experiments for classification and regression problems, using standard databases.