Model-Parallel Model Selection for Deep Learning Systems

TL;DR

This paper introduces Hydra, a novel framework for model parallelism in deep learning that improves efficiency by shard parallelism, enabling faster training of large models across multiple devices.

Contribution

The paper proposes a new shard parallelism approach called Hydra, combining task and model parallelism for more efficient multi-model training.

Findings

Hydra achieves significant speedups over traditional model parallelism.

Shard parallelism improves device utilization and training efficiency.

Framework enables training of larger models across multiple devices.

Abstract

As deep learning becomes more expensive, both in terms of time and compute, inefficiencies in machine learning (ML) training prevent practical usage of state-of-the-art models for most users. The newest model architectures are simply too large to be fit onto a single processor. To address the issue, many ML practitioners have turned to model parallelism as a method of distributing the computational requirements across several devices. Unfortunately, the sequential nature of neural networks causes very low efficiency and device utilization in model parallel training jobs. We propose a new form of "shard parallelism" combining task and model parallelism, then package it into a framework we name Hydra. Hydra recasts the problem of model parallelism in the multi-model context to produce a fine-grained parallel workload of independent model shards, rather than independent models. This new parallel design promises dramatic speedups relative to the traditional model parallelism paradigm.