Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment

TL;DR

This paper introduces a co-design framework for neural networks and deployment strategies that enhances real-time visual inference in IoVT systems by balancing throughput and accuracy on heterogeneous edge devices.

Contribution

It proposes a novel dynamic model structure and partitioning strategy, along with a multi-objective co-optimization approach, to improve inference performance on resource-constrained edge devices.

Findings

Achieved 12.05% and 18.83% throughput improvements on MNIST and ImageNet.

Demonstrated superior classification accuracy over baseline algorithms.

Ensured stable performance across different edge devices.

Abstract

As the volume of image data grows, data-oriented cloud computing in Internet of Video Things (IoVT) systems encounters latency issues. Task-oriented edge computing addresses this by shifting data analysis to the edge. However, limited computational power of edge devices poses challenges for executing visual tasks. Existing methods struggle to balance high model performance with low resource consumption; lightweight neural networks often underperform, while device-specific models designed by Neural Architecture Search (NAS) fail to adapt to heterogeneous devices. For these issues, we propose a novel co-design framework to optimize neural network architecture and deployment strategies during inference for high-throughput. Specifically, it implements a dynamic model structure based on re-parameterization, coupled with a Roofline-based model partitioning strategy to enhance the computational performance of edge devices. We also employ a multi-objective co-optimization approach to balance throughput and accuracy. Additionally, we derive mathematical consistency and convergence of partitioned models. Experimental results demonstrate significant improvements in throughput (12.05\% on MNIST, 18.83\% on ImageNet) and superior classification accuracy compared to baseline algorithms. Our method consistently achieves stable performance across different devices, underscoring its adaptability. Simulated experiments further confirm its efficacy in high-accuracy, real-time detection for small objects in IoVT systems.