Stochastic Particle Gradient Descent for Infinite Ensembles

TL;DR

This paper introduces a stochastic particle gradient descent method for infinite ensembles that effectively handles $L^1$-constraints in a rigorous way, enabling the construction of residual-type networks with proven convergence properties.

Contribution

It proposes a novel stochastic optimization approach for learning probability measures in ensemble models, overcoming regularization challenges and providing convergence guarantees.

Findings

The method can handle $L^1$-constraints rigorously.

It achieves convergence rates comparable to finite-dimensional stochastic optimization.

The resulting ensemble forms a residual-type network with an interior optimality property.

Abstract

The superior performance of ensemble methods with infinite models are well known. Most of these methods are based on optimization problems in infinite-dimensional spaces with some regularization, for instance, boosting methods and convex neural networks use $L^1$-regularization with the non-negative constraint. However, due to the difficulty of handling $L^1$-regularization, these problems require early stopping or a rough approximation to solve it inexactly. In this paper, we propose a new ensemble learning method that performs in a space of probability measures, that is, our method can handle the $L^1$-constraint and the non-negative constraint in a rigorous way. Such an optimization is realized by proposing a general purpose stochastic optimization method for learning probability measures via parameterization using transport maps on base models. As a result of running the method, a transport map to output an infinite ensemble is obtained, which forms a residual-type network. From the perspective of functional gradient methods, we give a convergence rate as fast as that of a stochastic optimization method for finite dimensional nonconvex problems. Moreover, we show an interior optimality property of a local optimality condition used in our analysis.