Learning (With) Distributed Optimization

TL;DR

This paper reviews the evolution of distributed optimization methods from early duality techniques to modern algorithms like ALADIN, emphasizing its advantages in non-convex problems and its historical significance.

Contribution

It provides a comprehensive overview of distributed optimization history and highlights ALADIN's unique features and potential in non-convex optimization.

Findings

ALADIN offers convergence guarantees for non-convex problems.

Historical progression from duality methods to ALADIN.

ADMM's practical effectiveness in machine learning and imaging.

Abstract

This paper provides an overview of the historical progression of distributed optimization techniques, tracing their development from early duality-based methods pioneered by Dantzig, Wolfe, and Benders in the 1960s to the emergence of the Augmented Lagrangian Alternating Direction Inexact Newton (ALADIN) algorithm. The initial focus on Lagrangian relaxation for convex problems and decomposition strategies led to the refinement of methods like the Alternating Direction Method of Multipliers (ADMM). The resurgence of interest in distributed optimization in the late 2000s, particularly in machine learning and imaging, demonstrated ADMM's practical efficacy and its unifying potential. This overview also highlights the emergence of the proximal center method and its applications in diverse domains. Furthermore, the paper underscores the distinctive features of ALADIN, which offers convergence guarantees for non-convex scenarios without introducing auxiliary variables, differentiating it from traditional augmentation techniques. In essence, this work encapsulates the historical trajectory of distributed optimization and underscores the promising prospects of ALADIN in addressing non-convex optimization challenges.