SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems

TL;DR

SLIDE introduces a CPU-based algorithm that combines randomized methods and workload optimization, outperforming GPU-based training in large-scale deep learning tasks without specialized hardware.

Contribution

The paper presents SLIDE, a novel algorithm that leverages smart randomized techniques and multi-core parallelism to significantly accelerate deep learning training on CPUs, reducing reliance on expensive hardware.

Findings

SLIDE trains large neural networks 3.5 times faster than TensorFlow on GPUs.

SLIDE achieves over 10x speedup over TensorFlow on the same CPU hardware.

Training with SLIDE on CPUs outperforms GPU-based training at all accuracy levels.

Abstract

Deep Learning (DL) algorithms are the central focus of modern machine learning systems. As data volumes keep growing, it has become customary to train large neural networks with hundreds of millions of parameters to maintain enough capacity to memorize these volumes and obtain state-of-the-art accuracy. To get around the costly computations associated with large models and data, the community is increasingly investing in specialized hardware for model training. However, specialized hardware is expensive and hard to generalize to a multitude of tasks. The progress on the algorithmic front has failed to demonstrate a direct advantage over powerful hardware such as NVIDIA-V100 GPUs. This paper provides an exception. We propose SLIDE (Sub-LInear Deep learning Engine) that uniquely blends smart randomized algorithms, with multi-core parallelism and workload optimization. Using just a CPU, SLIDE drastically reduces the computations during both training and inference outperforming an optimized implementation of Tensorflow (TF) on the best available GPU. Our evaluations on industry-scale recommendation datasets, with large fully connected architectures, show that training with SLIDE on a 44 core CPU is more than 3.5 times (1 hour vs. 3.5 hours) faster than the same network trained using TF on Tesla V100 at any given accuracy level. On the same CPU hardware, SLIDE is over 10x faster than TF. We provide codes and scripts for reproducibility.