# Tuple-oriented Compression for Large-scale Mini-batch Stochastic Gradient Descent

## TL;DR

This paper introduces tuple-oriented compression (TOC), a lossless scheme tailored for mini-batch stochastic gradient descent that significantly reduces data size and accelerates training in machine learning workloads.

## Contribution

It proposes a novel compression scheme specifically designed for MGD, along with new matrix operation techniques that operate directly on compressed data.

## Key findings

- Achieves up to 51x compression ratios
- Reduces MGD runtimes by up to 10.2x
- Improves efficiency of ML training workloads

## Abstract

Data compression is a popular technique for improving the efficiency of data processing workloads such as SQL queries and more recently, machine learning (ML) with classical batch gradient methods. But the efficacy of such ideas for mini-batch stochastic gradient descent (MGD), arguably the workhorse algorithm of modern ML, is an open question. MGD's unique data access pattern renders prior art, including those designed for batch gradient methods, less effective. We fill this crucial research gap by proposing a new lossless compression scheme we call tuple-oriented compression (TOC) that is inspired by an unlikely source, the string/text compression scheme Lempel-Ziv-Welch, but tailored to MGD in a way that preserves tuple boundaries within mini-batches. We then present a suite of novel compressed matrix operation execution techniques tailored to the TOC compression scheme that operate directly over the compressed data representation and avoid decompression overheads. An extensive empirical evaluation with real-world datasets shows that TOC consistently achieves substantial compression ratios by up to 51x and reduces runtimes for MGD workloads by up to 10.2x in popular ML systems.