Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference

TL;DR

Galaxy is a collaborative edge AI system that efficiently accelerates Transformer inference by leveraging idle resources across heterogeneous edge devices, significantly reducing latency and addressing privacy concerns.

Contribution

The paper introduces Galaxy, a novel resource-sharing framework with hybrid model parallelism and communication-computation overlap for efficient in-situ Transformer inference at the edge.

Findings

Achieves up to 2.5x latency reduction compared to state-of-the-art methods.

Effectively utilizes idle resources across heterogeneous edge devices.

Demonstrates significant performance improvements in various edge environments.

Abstract

Transformer-based models have unlocked a plethora of powerful intelligent applications at the edge, such as voice assistant in smart home. Traditional deployment approaches offload the inference workloads to the remote cloud server, which would induce substantial pressure on the backbone network as well as raise users' privacy concerns. To address that, in-situ inference has been recently recognized for edge intelligence, but it still confronts significant challenges stemming from the conflict between intensive workloads and limited on-device computing resources. In this paper, we leverage our observation that many edge environments usually comprise a rich set of accompanying trusted edge devices with idle resources and propose Galaxy, a collaborative edge AI system that breaks the resource walls across heterogeneous edge devices for efficient Transformer inference acceleration. Galaxy introduces a novel hybrid model parallelism to orchestrate collaborative inference, along with a heterogeneity-aware parallelism planning for fully exploiting the resource potential. Furthermore, Galaxy devises a tile-based fine-grained overlapping of communication and computation to mitigate the impact of tensor synchronizations on inference latency under bandwidth-constrained edge environments. Extensive evaluation based on prototype implementation demonstrates that Galaxy remarkably outperforms state-of-the-art approaches under various edge environment setups, achieving up to 2.5x end-to-end latency reduction.