Hyperion: Low-Latency Ultra-HD Video Analytics via Collaborative Vision Transformer Inference

TL;DR

Hyperion is a collaborative cloud-device framework that enables low-latency Ultra-HD video analytics using vision transformers, balancing computational load and transmission to improve speed and accuracy in dynamic network conditions.

Contribution

Hyperion introduces a novel collaborative approach with importance scoring, adaptive scheduling, and result fusion to optimize Ultra-HD transformer inference across cloud and device.

Findings

Increases frame processing rate by up to 1.61x

Improves accuracy by up to 20.2%

Effective under various network conditions

Abstract

Recent advancements in array-camera videography enable real-time capturing of ultra-high-definition (Ultra-HD) videos, providing rich visual information in a large field of view. However, promptly processing such data using state-of-the-art transformer-based vision foundation models faces significant computational overhead in on-device computing or transmission overhead in cloud computing. In this paper, we present Hyperion, the first cloud-device collaborative framework that enables low-latency inference on Ultra-HD vision data using off-the-shelf vision transformers over dynamic networks. Hyperion addresses the computational and transmission bottleneck of Ultra-HD vision transformers by exploiting the intrinsic property in vision Transformer models. Specifically, Hyperion integrates a collaboration-aware importance scorer that identifies critical regions at the patch level, a dynamic scheduler that adaptively adjusts patch transmission quality to balance latency and accuracy under dynamic network conditions, and a weighted ensembler that fuses edge and cloud results to improve accuracy. Experimental results demonstrate that Hyperion enhances frame processing rate by up to 1.61 times and improves the accuracy by up to 20.2% when compared with state-of-the-art baselines under various network environments.