Harnessing the Power of Serverless Runtimes for Large-Scale Optimization

TL;DR

This paper demonstrates how serverless runtimes like AWS Lambda can be effectively utilized for large-scale optimization tasks, achieving significant speedups and efficiencies in a master-worker setup.

Contribution

It introduces a novel approach to using serverless platforms for generic optimization problems, including implementation details and analysis of performance bottlenecks.

Findings

Speedups up to 256 workers

Efficiency above 70% with 64 workers

Identification of algorithmic and system bottlenecks

Abstract

The event-driven and elastic nature of serverless runtimes makes them a very efficient and cost-effective alternative for scaling up computations. So far, they have mostly been used for stateless, data parallel and ephemeral computations. In this work, we propose using serverless runtimes to solve generic, large-scale optimization problems. Specifically, we build a master-worker setup using AWS Lambda as the source of our workers, implement a parallel optimization algorithm to solve a regularized logistic regression problem, and show that relative speedups up to 256 workers and efficiencies above 70% up to 64 workers can be expected. We also identify possible algorithmic and system-level bottlenecks, propose improvements, and discuss the limitations and challenges in realizing these improvements.