MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes

TL;DR

This paper advances federated optimization by extending the MARINA-P algorithm to non-smooth convex problems, providing optimal convergence rates and communication complexity bounds, and demonstrating superior empirical performance with adaptive stepsizes.

Contribution

It introduces the first theoretical analysis of distributed non-smooth convex federated optimization with server-to-worker compression, extending MARINA-P and EF21-P algorithms.

Findings

MARINA-P with correlated compressors outperforms other methods.

Achieves optimal $O(1/\sqrt{T})$ convergence rate.

Provides comprehensive analysis for various stepsize schemes.

Abstract

Non-smooth communication-efficient federated optimization is crucial for many machine learning applications, yet remains largely unexplored theoretically. Recent advancements have primarily focused on smooth convex and non-convex regimes, leaving a significant gap in understanding the non-smooth convex setting. Additionally, existing literature often overlooks efficient server-to-worker communication (downlink), focusing primarily on worker-to-server communication (uplink). We consider a setup where uplink costs are negligible and focus on optimizing downlink communication by improving state-of-the-art schemes like EF21-P (arXiv:2209.15218) and MARINA-P (arXiv:2402.06412) in the non-smooth convex setting. We extend the non-smooth convex theory of EF21-P [Anonymous, 2024], originally developed for single-node scenarios, to the distributed setting, and extend MARINA-P to the non-smooth convex setting. For both algorithms, we prove an optimal $O(1/\sqrt{T})$ convergence rate and establish communication complexity bounds matching classical subgradient methods. We provide theoretical guarantees under constant, decreasing, and adaptive (Polyak-type) stepsizes. Our experiments demonstrate that MARINA-P with correlated compressors outperforms other methods in both smooth non-convex and non-smooth convex settings. This work presents the first theoretical results for distributed non-smooth optimization with server-to-worker compression, along with comprehensive analysis for various stepsize schemes.