On Acceleration of Gradient-Based Empirical Risk Minimization using Local Polynomial Regression

TL;DR

This paper introduces accelerated algorithms based on local polynomial interpolation for empirical risk minimization, demonstrating improved theoretical complexity and empirical performance over traditional gradient methods in certain settings.

Contribution

It proposes two accelerated methods for ERM using LPI-GD, achieving better oracle complexity and providing the first empirical evaluation of local polynomial interpolation-based gradient methods.

Findings

Accelerated methods reduce oracle complexity to rom or LPI-GD.

Empirical results show LPI-GD outperforms GD and SGD in some scenarios.

Theoretical analysis confirms acceleration benefits in specific parameter regimes.

Abstract

We study the acceleration of the Local Polynomial Interpolation-based Gradient Descent method (LPI-GD) recently proposed for the approximate solution of empirical risk minimization problems (ERM). We focus on loss functions that are strongly convex and smooth with condition number $\sigma$. We additionally assume the loss function is $\eta$-H\"older continuous with respect to the data. The oracle complexity of LPI-GD is $\tilde{O}\left(\sigma m^d \log(1/\varepsilon)\right)$ for a desired accuracy $\varepsilon$, where $d$ is the dimension of the parameter space, and $m$ is the cardinality of an approximation grid. The factor $m^d$ can be shown to scale as $O((1/\varepsilon)^{d/2\eta})$. LPI-GD has been shown to have better oracle complexity than gradient descent (GD) and stochastic gradient descent (SGD) for certain parameter regimes. We propose two accelerated methods for the ERM problem based on LPI-GD and show an oracle complexity of $\tilde{O}\left(\sqrt{\sigma} m^d \log(1/\varepsilon)\right)$. Moreover, we provide the first empirical study on local polynomial interpolation-based gradient methods and corroborate that LPI-GD has better performance than GD and SGD in some scenarios, and the proposed methods achieve acceleration.