SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization

TL;DR

SDNA is a new stochastic dual Newton ascent algorithm that efficiently leverages curvature information for empirical risk minimization, offering significant theoretical and practical improvements over existing methods.

Contribution

The paper introduces SDNA, a dual-based stochastic optimization method that utilizes full curvature information, outperforming traditional stochastic dual coordinate ascent algorithms.

Findings

SDNA achieves faster convergence than existing methods.

Utilizes all curvature information for improved optimization.

Applicable to quadratic loss functions with enhanced performance.

Abstract

We propose a new algorithm for minimizing regularized empirical loss: Stochastic Dual Newton Ascent (SDNA). Our method is dual in nature: in each iteration we update a random subset of the dual variables. However, unlike existing methods such as stochastic dual coordinate ascent, SDNA is capable of utilizing all curvature information contained in the examples, which leads to striking improvements in both theory and practice - sometimes by orders of magnitude. In the special case when an L2-regularizer is used in the primal, the dual problem is a concave quadratic maximization problem plus a separable term. In this regime, SDNA in each step solves a proximal subproblem involving a random principal submatrix of the Hessian of the quadratic function; whence the name of the method. If, in addition, the loss functions are quadratic, our method can be interpreted as a novel variant of the recently introduced Iterative Hessian Sketch.