Conditions for Convergence in Regularized Machine Learning Objectives

TL;DR

This paper examines the convergence conditions of regularized convex optimization algorithms in distributed machine learning, highlighting theoretical bounds and practical implications for practitioners.

Contribution

It provides a comprehensive analysis of convergence conditions and lower bounds for distributed convex optimization algorithms, addressing the impact of communication costs.

Findings

Convergence can be established despite communication slowdowns.

Lower bounds on convergence rates are derived for distributed settings.

Practical guidelines for ensuring convergence in distributed ML are discussed.

Abstract

Analysis of the convergence rates of modern convex optimization algorithms can be achived through binary means: analysis of emperical convergence, or analysis of theoretical convergence. These two pathways of capturing information diverge in efficacy when moving to the world of distributed computing, due to the introduction of non-intuitive, non-linear slowdowns associated with broadcasting, and in some cases, gathering operations. Despite these nuances in the rates of convergence, we can still show the existence of convergence, and lower bounds for the rates. This paper will serve as a helpful cheat-sheet for machine learning practitioners encountering this problem class in the field.