Batching-Aware Joint Model Onloading and Offloading for Hierarchical Multi-Task Inference

TL;DR

This paper introduces J3O, a unified framework for optimizing multi-task model deployment and query routing in hierarchical edge systems, effectively balancing accuracy, memory, and communication constraints.

Contribution

It proposes a novel joint optimization algorithm for onloading and offloading multi-task models, extended to include batching, with scalable solutions for resource-constrained hierarchical inference.

Findings

Achieves over 97% of optimal accuracy in experiments.

Runs less than 15% of the time of the optimal solver.

Effectively handles multi-task, hierarchical inference scenarios.

Abstract

The growing demand for intelligent services on resource-constrained edge devices has spurred the development of collaborative inference systems that distribute workloads across end devices, edge servers, and the cloud. While most existing frameworks focus on single-task, single-model scenarios, many real-world applications (e.g., autonomous driving and augmented reality) require concurrent execution of diverse tasks including detection, segmentation, and depth estimation. In this work, we propose a unified framework to jointly decide which multi-task models to deploy (onload) at clients and edge servers, and how to route queries across the hierarchy (offload) to maximize overall inference accuracy under memory, compute, and communication constraints. We formulate this as a mixed-integer program and introduce J3O (Joint Optimization of Onloading and Offloading), an alternating algorithm that (i) greedily selects models to onload via Lagrangian-relaxed submodular optimization and (ii) determines optimal offloading via constrained linear programming. We further extend J3O to account for batching at the edge, maintaining scalability under heterogeneous task loads. Experiments show J3O consistently achieves over $97\%$ of the optimal accuracy while incurring less than $15\%$ of the runtime required by the optimal solver across multi-task benchmarks.