Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimization

TL;DR

COACH is a framework that optimizes end-cloud collaborative inference by minimizing pipeline bubbles, resulting in significantly faster inference and higher throughput while maintaining accuracy.

Contribution

The paper introduces COACH, a novel near bubble-free pipeline optimization framework with offline and online components for improved DNN inference in end-cloud collaboration.

Findings

Up to 1.7x faster inference compared to baselines.

Achieves 2.1x higher system throughput.

Maintains comparable accuracy with improved efficiency.

Abstract

End-cloud collaboration offers a promising strategy to enhance the Quality of Service (QoS) in DNN inference by offloading portions of the inference workload from end devices to cloud servers. Despite the potential, the complex model architectures and dynamic network conditions will introduce numerous bubbles (\ie, idle waiting time) in pipeline execution, resulting in inefficient resource utilization and degraded QoS. To address these challenges, we introduce a novel framework named COACH, designed for near bubble-free pipeline collaborative inference, thereby achieving low inference latency and high system throughput. Initially, COACH employs an \textit{offline} component that utilizes an efficient recursive divide-and-conquer algorithm to optimize both model partitioning and transmission quantization, aiming to minimize the occurrence of pipeline bubbles. Subsequently, the \textit{online} component in COACH employs an adaptive quantization adjustment and a context-aware caching strategy to further stabilize pipeline execution. Specifically, COACH analyzes the correlation between intermediate data and label semantic centers in the cache, along with its influence on the quantization adjustment, thereby effectively accommodating network fluctuations. Our experiments demonstrate the efficacy of COACH in reducing inference latency and enhancing system throughput. Notably, while maintaining comparable accuracy, COACH achieves up to 1.7x faster inference and 2.1x higher system throughput than baselines.