SCOPE: Scalable Composite Optimization for Learning on Spark

TL;DR

SCOPE is a scalable, efficient distributed optimization method implemented on Spark, designed for large-scale composite problems like logistic regression and SVM, with proven convergence and superior empirical performance.

Contribution

The paper introduces SCOPE, a novel distributed stochastic optimization algorithm for scalable learning on Spark, with theoretical convergence guarantees and improved empirical results.

Findings

SCOPE converges linearly for convex objectives.

SCOPE outperforms existing distributed methods on real datasets.

SCOPE is both computation- and communication-efficient.

Abstract

Many machine learning models, such as logistic regression~(LR) and support vector machine~(SVM), can be formulated as composite optimization problems. Recently, many distributed stochastic optimization~(DSO) methods have been proposed to solve the large-scale composite optimization problems, which have shown better performance than traditional batch methods. However, most of these DSO methods are not scalable enough. In this paper, we propose a novel DSO method, called \underline{s}calable \underline{c}omposite \underline{op}timization for l\underline{e}arning~({SCOPE}), and implement it on the fault-tolerant distributed platform \mbox{Spark}. SCOPE is both computation-efficient and communication-efficient. Theoretical analysis shows that SCOPE is convergent with linear convergence rate when the objective function is convex. Furthermore, empirical results on real datasets show that SCOPE can outperform other state-of-the-art distributed learning methods on Spark, including both batch learning methods and DSO methods.