FreeRide: Harvesting Bubbles in Pipeline Parallelism

TL;DR

FreeRide is a system that efficiently harvests idle GPU resources during pipeline parallelism in LLM training to run side tasks, reducing costs with minimal overhead.

Contribution

It introduces a novel system, FreeRide, that simplifies programming and manages GPU resources to harvest pipeline bubbles for side tasks during LLM training.

Findings

Achieves 7.8% average cost savings in LLM training.

Maintains about 1% overhead during training.

Supports diverse side tasks like graph analytics and image processing.

Abstract

The occurrence of bubbles in pipeline parallelism is an inherent limitation that can account for more than 40% of the large language model (LLM) training time and is one of the main reasons for the underutilization of GPU resources in LLM training. Harvesting these bubbles for GPU side tasks can increase resource utilization and reduce training costs but comes with challenges. First, because bubbles are discontinuous with various shapes, programming side tasks becomes difficult while requiring excessive engineering effort. Second, a side task can compete with pipeline training for GPU resources and incur significant overhead. To address these challenges, we propose FreeRide, a system designed to harvest bubbles in pipeline parallelism for side tasks. FreeRide provides programmers with interfaces to implement side tasks easily, manages bubbles and side tasks during pipeline training, and controls access to GPU resources by side tasks to reduce overhead. We demonstrate that FreeRide achieves 7.8% average cost savings with a negligible overhead of about 1% in training LLMs while serving model training, graph analytics, and image processing side tasks.