Distributed Block-diagonal Approximation Methods for Regularized Empirical Risk Minimization

TL;DR

This paper introduces a flexible distributed optimization framework for large-scale empirical risk minimization that converges globally and outperforms existing methods in speed and efficiency.

Contribution

It presents a unified dual-based approach for distributed ERM that allows approximate sub-problem solutions and applies broadly across machine learning tasks.

Findings

Achieves global linear convergence for a wide range of problems.

Demonstrates faster empirical convergence compared to existing methods.

Applicable to classification, regression, and structured prediction tasks.

Abstract

In recent years, there is a growing need to train machine learning models on a huge volume of data. Designing efficient distributed optimization algorithms for empirical risk minimization (ERM) has therefore become an active and challenging research topic. In this paper, we propose a flexible framework for distributed ERM training through solving the dual problem, which provides a unified description and comparison of existing methods. Our approach requires only approximate solutions of the sub-problems involved in the optimization process, and is versatile to be applied on many large-scale machine learning problems including classification, regression, and structured prediction. We show that our approach enjoys global linear convergence for a broader class of problems, and achieves faster empirical performance, compared with existing works.