ViTMAlis: Towards Latency-Critical Mobile Video Analytics with Vision Transformers

TL;DR

ViTMAlis is a framework that optimizes latency-critical mobile video analytics using vision transformers by dynamically balancing resolution, transmission, and inference to reduce delays and improve accuracy.

Contribution

It introduces a novel dynamic mixed-resolution inference strategy and a ViT-native offloading framework tailored for latency-critical dense prediction tasks.

Findings

Significantly reduces end-to-end offloading latency.

Improves user-perceived rendering accuracy.

Outperforms state-of-the-art latency-adaptive baselines.

Abstract

Edge-assisted mobile video analytics (MVA) applications are increasingly shifting from using vision models based on convolutional neural networks (CNNs) to those built on vision transformers (ViTs) to leverage their superior global context modeling and generalization capabilities. However, deploying these advanced models in latency-critical MVA scenarios presents significant challenges. Unlike traditional CNN-based offloading paradigms where network transmission is the primary bottleneck, ViT-based systems are constrained by substantial inference delays, particularly for dense prediction tasks where the need for high-resolution inputs exacerbates the inherent quadratic computational complexity of ViTs. To address these challenges, we propose a dynamic mixed-resolution inference strategy tailored for ViT-backboned dense prediction models, enabling flexible runtime trade-offs between speed and accuracy. Building on this, we introduce ViTMAlis, a ViT-native device-to-edge offloading framework that dynamically adapts to network conditions and video content to jointly reduce transmission and inference delays. We implement a fully functional prototype of ViTMAlis on commodity mobile and edge devices. Extensive experiments demonstrate that, compared to state-of-the-art accuracy-centric, content-aware, and latency-adaptive baselines, ViTMAlis significantly reduces end-to-end offloading latency while improving user-perceived rendering accuracy, providing a practical foundation for next-generation mobile intelligence.