CSG: A stochastic gradient method for a wide class of optimization problems appearing in a machine learning or data-driven context

TL;DR

This paper extends the CSG stochastic gradient method to broader optimization problems in machine learning by introducing new ways to compute integration weights, with proven convergence and demonstrated efficiency.

Contribution

It introduces alternative methods for calculating integration weights in CSG, expanding its applicability beyond simple probability distributions.

Findings

Enhanced scope of CSG for complex stochastic problems

Proven convergence of the new CSG variants

Demonstrated efficiency over classical methods

Abstract

A recent article introduced thecontinuous stochastic gradient method (CSG) for the efficient solution of a class of stochastic optimization problems. While the applicability of known stochastic gradient type methods is typically limited to expected risk functions, no such limitation exists for CSG. This advantage stems from the computation of design dependent integration weights, allowing for optimal usage of available information and therefore stronger convergence properties. However, the nature of the formula used for these integration weights essentially limited the practical applicability of this method to problems in which stochasticity enters via a low-dimensional and sufficiently simple probability distribution. In this paper we significantly extend the scope of the CSG method by presenting alternative ways to calculate the integration weights. A full convergence analysis for this new variant of the CSG method is presented and its efficiency is demonstrated in comparison to more classical stochastic gradient methods by means of a number of problem classes relevant to stochastic optimization and machine learning.