Breaking the accuracy-resource dilemma: a lightweight adaptive video inference enhancement

TL;DR

This paper introduces a lightweight, adaptive video inference framework that dynamically balances resource use and performance by switching models based on real-time conditions, leveraging spatiotemporal correlations.

Contribution

It proposes a novel fuzzy controller-guided framework for real-time adaptive model switching in video inference, optimizing resource efficiency and accuracy.

Findings

Effectively balances resource utilization and inference performance.

Dynamically switches models based on real-time resource conditions.

Leverages spatiotemporal correlations for improved inference.

Abstract

Existing video inference (VI) enhancement methods typically aim to improve performance by scaling up model sizes and employing sophisticated network architectures. While these approaches demonstrated state-of-the-art performance, they often overlooked the trade-off of resource efficiency and inference effectiveness, leading to inefficient resource utilization and suboptimal inference performance. To address this problem, a fuzzy controller (FC-r) is developed based on key system parameters and inference-related metrics. Guided by the FC-r, a VI enhancement framework is proposed, where the spatiotemporal correlation of targets across adjacent video frames is leveraged. Given the real-time resource conditions of the target device, the framework can dynamically switch between models of varying scales during VI. Experimental results demonstrate that the proposed method effectively achieves a balance between resource utilization and inference performance.