Elastic CoCoA: Scaling In to Improve Convergence

TL;DR

This paper introduces Elastic CoCoA, a framework that dynamically adjusts the number of workers during training to optimize convergence rate, significantly reducing training time across multiple datasets.

Contribution

We propose Chicle, an elastic framework that adaptively tunes the number of workers in CoCoA to improve convergence speed and automate optimal resource allocation.

Findings

Chicle accelerates training by up to 5.96x compared to static configurations.

The optimal number of workers varies during training, and Chicle adapts effectively.

Chicle reliably finds near-optimal worker settings in most cases.

Abstract

In this paper we experimentally analyze the convergence behavior of CoCoA and show, that the number of workers required to achieve the highest convergence rate at any point in time, changes over the course of the training. Based on this observation, we build Chicle, an elastic framework that dynamically adjusts the number of workers based on feedback from the training algorithm, in order to select the number of workers that results in the highest convergence rate. In our evaluation of 6 datasets, we show that Chicle is able to accelerate the time-to-accuracy by a factor of up to 5.96x compared to the best static setting, while being robust enough to find an optimal or near-optimal setting automatically in most cases.