Tuning the Scheduling of Distributed Stochastic Gradient Descent with Bayesian Optimization

TL;DR

This paper introduces a Bayesian optimization-based method to efficiently tune distributed SGD system parameters, achieving faster convergence and better configurations in high-dimensional settings.

Contribution

It develops a probabilistic model that simulates distributed SGD behavior, enabling rapid tuning in high-dimensional parameter spaces.

Findings

Optimizer converges within ten iterations.

Outperforms generic optimizers by up to 2X.

Handles over thirty parameters effectively.

Abstract

We present an optimizer which uses Bayesian optimization to tune the system parameters of distributed stochastic gradient descent (SGD). Given a specific context, our goal is to quickly find efficient configurations which appropriately balance the load between the available machines to minimize the average SGD iteration time. Our experiments consider setups with over thirty parameters. Traditional Bayesian optimization, which uses a Gaussian process as its model, is not well suited to such high dimensional domains. To reduce convergence time, we exploit the available structure. We design a probabilistic model which simulates the behavior of distributed SGD and use it within Bayesian optimization. Our model can exploit many runtime measurements for inference per evaluation of the objective function. Our experiments show that our resulting optimizer converges to efficient configurations within ten iterations, the optimized configurations outperform those found by generic optimizer in thirty iterations by up to 2X.