# Large-Scale Stochastic Learning using GPUs

## TL;DR

This paper presents a GPU-accelerated stochastic learning system capable of handling large-scale datasets efficiently, achieving significant speed-ups through parallelization and distributed training techniques.

## Contribution

It introduces a GPU implementation of stochastic coordinate descent and a novel method for aggregating updates in distributed settings, enabling scalable, fast training on massive datasets.

## Key findings

- Up to 35x speed-up over CPU implementations.
- Scales out across 8 nodes with minimal loss in training time.
- Trains on 200 million examples in around 4 seconds using 4 GPUs.

## Abstract

In this work we propose an accelerated stochastic learning system for very large-scale applications. Acceleration is achieved by mapping the training algorithm onto massively parallel processors: we demonstrate a parallel, asynchronous GPU implementation of the widely used stochastic coordinate descent/ascent algorithm that can provide up to 35x speed-up over a sequential CPU implementation. In order to train on very large datasets that do not fit inside the memory of a single GPU, we then consider techniques for distributed stochastic learning. We propose a novel method for optimally aggregating model updates from worker nodes when the training data is distributed either by example or by feature. Using this technique, we demonstrate that one can scale out stochastic learning across up to 8 worker nodes without any significant loss of training time. Finally, we combine GPU acceleration with the optimized distributed method to train on a dataset consisting of 200 million training examples and 75 million features. We show by scaling out across 4 GPUs, one can attain a high degree of training accuracy in around 4 seconds: a 20x speed-up in training time compared to a multi-threaded, distributed implementation across 4 CPUs.