Adaptive Sequential Optimization with Applications to Machine Learning

TL;DR

This paper presents an adaptive framework for sequentially solving slowly changing optimization problems in machine learning, ensuring controlled excess risk by dynamically adjusting sample sizes based on estimated minimizer changes.

Contribution

It introduces a novel adaptive method for selecting sample sizes in sequential optimization, accounting for slow changes in problem minimizers, applicable to regression and classification tasks.

Findings

Method effectively controls excess risk in experiments.

Adaptive approach outperforms fixed-sample strategies.

Works well with synthetic and real data.

Abstract

A framework is introduced for solving a sequence of slowly changing optimization problems, including those arising in regression and classification applications, using optimization algorithms such as stochastic gradient descent (SGD). The optimization problems change slowly in the sense that the minimizers change at either a fixed or bounded rate. A method based on estimates of the change in the minimizers and properties of the optimization algorithm is introduced for adaptively selecting the number of samples needed from the distributions underlying each problem in order to ensure that the excess risk, i.e., the expected gap between the loss achieved by the approximate minimizer produced by the optimization algorithm and the exact minimizer, does not exceed a target level. Experiments with synthetic and real data are used to confirm that this approach performs well.