A Stochastic Large-scale Machine Learning Algorithm for Distributed Features and Observations

TL;DR

This paper introduces a stochastic distributed optimization algorithm capable of handling both features and observations in large-scale machine learning, with proven convergence and improved early performance in Spark-based experiments.

Contribution

It presents a novel stochastic algorithm for double distributed data, addressing a gap in existing methods that only handle either features or observations.

Findings

Convergence established under various learning rate conditions

Superior early iteration performance in Spark experiments

Effective handling of both features and observations in distributed setting

Abstract

As the size of modern data sets exceeds the disk and memory capacities of a single computer, machine learning practitioners have resorted to parallel and distributed computing. Given that optimization is one of the pillars of machine learning and predictive modeling, distributed optimization methods have recently garnered ample attention, in particular when either observations or features are distributed, but not both. We propose a general stochastic algorithm where observations, features, and gradient components can be sampled in a double distributed setting, i.e., with both features and observations distributed. Very technical analyses establish convergence properties of the algorithm under different conditions on the learning rate (diminishing to zero or constant). Computational experiments in Spark demonstrate a superior performance of our algorithm versus a benchmark in early iterations of the algorithm, which is due to the stochastic components of the algorithm.