HAMSI: A Parallel Incremental Optimization Algorithm Using Quadratic Approximations for Solving Partially Separable Problems

TL;DR

HAMSI is a parallel, second-order optimization algorithm that uses quadratic approximations to efficiently solve large-scale partially separable problems, outperforming stochastic gradient descent in convergence speed.

Contribution

The paper introduces HAMSI, a provably convergent, parallel second-order method utilizing quadratic approximations for large-scale optimization, with demonstrated superior convergence over stochastic gradient descent.

Findings

HAMSI converges faster than parallel stochastic gradient descent.

HAMSI scales well with the number of processors.

Memory usage of HAMSI scales linearly with problem size.

Abstract

We propose HAMSI (Hessian Approximated Multiple Subsets Iteration), which is a provably convergent, second order incremental algorithm for solving large-scale partially separable optimization problems. The algorithm is based on a local quadratic approximation, and hence, allows incorporating curvature information to speed-up the convergence. HAMSI is inherently parallel and it scales nicely with the number of processors. Combined with techniques for effectively utilizing modern parallel computer architectures, we illustrate that the proposed method converges more rapidly than a parallel stochastic gradient descent when both methods are used to solve large-scale matrix factorization problems. This performance gain comes only at the expense of using memory that scales linearly with the total size of the optimization variables. We conclude that HAMSI may be considered as a viable alternative in many large scale problems, where first order methods based on variants of stochastic gradient descent are applicable.