Guided Learning of Nonconvex Models through Successive Functional Gradient Optimization

TL;DR

This paper introduces a framework for training nonconvex models like neural networks using successive functional gradient optimization, combining mirror descent in function space with theoretical and empirical validation.

Contribution

It proposes a novel training framework based on successive functional gradient optimization and mirror descent for nonconvex models, with theoretical analysis and empirical evidence of improved performance.

Findings

Better performance than standard training methods

Theoretical guarantees for convergence

Empirical validation on neural network training

Abstract

This paper presents a framework of successive functional gradient optimization for training nonconvex models such as neural networks, where training is driven by mirror descent in a function space. We provide a theoretical analysis and empirical study of the training method derived from this framework. It is shown that the method leads to better performance than that of standard training techniques.