CoCoA: A General Framework for Communication-Efficient Distributed Optimization

TL;DR

CoCoA is a versatile framework for communication-efficient distributed optimization that effectively handles various regularizers and loss functions, demonstrating superior performance on real datasets.

Contribution

The paper introduces CoCoA, a general framework for distributed optimization with improved communication efficiency and convergence guarantees for a wide range of regularizers and loss functions.

Findings

Outperforms state-of-the-art methods in distributed settings

Handles non-strongly-convex regularizers like L1 and elastic net

Provides convergence guarantees for convex regularized objectives

Abstract

The scale of modern datasets necessitates the development of efficient distributed optimization methods for machine learning. We present a general-purpose framework for distributed computing environments, CoCoA, that has an efficient communication scheme and is applicable to a wide variety of problems in machine learning and signal processing. We extend the framework to cover general non-strongly-convex regularizers, including L1-regularized problems like lasso, sparse logistic regression, and elastic net regularization, and show how earlier work can be derived as a special case. We provide convergence guarantees for the class of convex regularized loss minimization objectives, leveraging a novel approach in handling non-strongly-convex regularizers and non-smooth loss functions. The resulting framework has markedly improved performance over state-of-the-art methods, as we illustrate with an extensive set of experiments on real distributed datasets.