Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds

TL;DR

This paper develops a comprehensive theoretical framework for distributed sketching in randomized optimization, providing new bounds, unbiased methods, and practical insights for large-scale, privacy-preserving, and resilient distributed systems.

Contribution

It introduces novel approximation guarantees, tight concentration bounds, and unbiased parameter averaging techniques for sketched second-order optimization in distributed settings.

Findings

New bounds for classical sketching methods

Unbiased parameter averaging algorithms for distributed Hessian sketching

Experimental validation on large-scale cloud platform

Abstract

We consider distributed optimization methods for problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We leverage randomized sketches for reducing the problem dimensions as well as preserving privacy and improving straggler resilience in asynchronous distributed systems. We derive novel approximation guarantees for classical sketching methods and establish tight concentration results that serve as both upper and lower bounds on the error. We then extend our analysis to the accuracy of parameter averaging for distributed sketches. Furthermore, we develop unbiased parameter averaging methods for randomized second order optimization for regularized problems that employ sketching of the Hessian. Existing works do not take the bias of the estimators into consideration, which limits their application to massively parallel computation. We provide closed-form formulas for regularization parameters and step sizes that provably minimize the bias for sketched Newton directions. Additionally, we demonstrate the implications of our theoretical findings via large scale experiments on a serverless cloud computing platform.