Principled Curriculum Learning using Parameter Continuation Methods

TL;DR

This paper introduces a parameter continuation approach for neural network optimization, linking homotopies and curriculum learning, which improves generalization over existing methods like ADAM in supervised and unsupervised tasks.

Contribution

It presents a theoretically justified parameter continuation method that enhances neural network training and generalization, bridging curriculum learning and homotopy techniques.

Findings

Outperforms ADAM in generalization on various tasks

Theoretically grounded approach improves neural network training

Effective for both supervised and unsupervised learning

Abstract

In this work, we propose a parameter continuation method for the optimization of neural networks. There is a close connection between parameter continuation, homotopies, and curriculum learning. The methods we propose here are theoretically justified and practically effective for several problems in deep neural networks. In particular, we demonstrate better generalization performance than state-of-the-art optimization techniques such as ADAM for supervised and unsupervised learning tasks.