Adaptive Configuration Selection for Multi-Model Inference Pipelines in Edge Computing

TL;DR

This paper presents a reinforcement learning-based framework for configuring multi-model inference pipelines on edge devices, optimizing QoS and costs while respecting resource constraints.

Contribution

It introduces a novel decision-making model and an RL-based algorithm for adaptive pipeline configuration considering resource limitations.

Findings

Significantly improves QoS in real Kubernetes experiments

Reduces costs compared to baseline algorithms

Speeds up decision-making for complex pipelines

Abstract

The growing demand for real-time processing tasks is driving the need for multi-model inference pipelines on edge devices. However, cost-effectively deploying these pipelines while optimizing Quality of Service (QoS) and costs poses significant challenges. Existing solutions often neglect device resource constraints, focusing mainly on inference accuracy and cost efficiency. To address this, we develop a framework for configuring multi-model inference pipelines. Specifically: 1) We model the decision-making problem by considering the pipeline's QoS, costs, and device resource limitations. 2) We create a feature extraction module using residual networks and a load prediction model based on Long Short-Term Memory (LSTM) to gather comprehensive node and pipeline status information. Then, we implement a Reinforcement Learning (RL) algorithm based on policy gradients for online configuration decisions. 3) Experiments conducted in a real Kubernetes cluster show that our approach significantly improve QoS while reducing costs and shorten decision-making time for complex pipelines compared to baseline algorithms.