A Class of Parallel Doubly Stochastic Algorithms for Large-Scale Learning

TL;DR

This paper introduces RAPSA, a novel parallel stochastic algorithm for large-scale learning that operates on randomly selected data blocks and training subsets, with proven convergence and extensions for asynchronous updates.

Contribution

The paper presents RAPSA, the first method combining parallelism in both data and feature block selection, with convergence guarantees for convex objectives and an accelerated variant.

Findings

RAPSA converges to the optimal classifier for convex problems.

The accelerated version ARAPSA improves convergence speed.

Algorithms perform well on large-scale linear and image classification tasks.

Abstract

We consider learning problems over training sets in which both, the number of training examples and the dimension of the feature vectors, are large. To solve these problems we propose the random parallel stochastic algorithm (RAPSA). We call the algorithm random parallel because it utilizes multiple parallel processors to operate on a randomly chosen subset of blocks of the feature vector. We call the algorithm stochastic because processors choose training subsets uniformly at random. Algorithms that are parallel in either of these dimensions exist, but RAPSA is the first attempt at a methodology that is parallel in both the selection of blocks and the selection of elements of the training set. In RAPSA, processors utilize the randomly chosen functions to compute the stochastic gradient component associated with a randomly chosen block. The technical contribution of this paper is to show that this minimally coordinated algorithm converges to the optimal classifier when the training objective is convex. Moreover, we present an accelerated version of RAPSA (ARAPSA) that incorporates the objective function curvature information by premultiplying the descent direction by a Hessian approximation matrix. We further extend the results for asynchronous settings and show that if the processors perform their updates without any coordination the algorithms are still convergent to the optimal argument. RAPSA and its extensions are then numerically evaluated on a linear estimation problem and a binary image classification task using the MNIST handwritten digit dataset.