SneakPeek: Data-Aware Model Selection and Scheduling for Inference Serving on the Edge

TL;DR

SneakPeek introduces a data-aware model selection and scheduling method for edge inference systems, improving efficiency and accuracy without hardware scaling by dynamically estimating model accuracy and batching in constrained environments.

Contribution

The paper presents a novel accuracy scaling approach using ML models to dynamically estimate model accuracy and incorporates batching and priority-based scheduling for edge inference systems.

Findings

Higher-utility schedules achieved in real-world applications.

Improved inference accuracy in resource-constrained environments.

Effective batching reduces GPU overhead.

Abstract

Modern applications increasingly rely on inference serving systems to provide low-latency insights with a diverse set of machine learning models. Existing systems often utilize resource elasticity to scale with demand. However, many applications cannot rely on hardware scaling when deployed at the edge or other resource-constrained environments. In this work, we propose a model selection and scheduling algorithm that implements accuracy scaling to increase efficiency for these more constrained deployments. We show that existing schedulers that make decisions using profiled model accuracy are biased toward the label distribution present in the test dataset. To address this problem, we propose using ML models -- which we call SneakPeek models -- to dynamically adjust estimates of model accuracy, based on the underlying data. Furthermore, we greedily incorporate inference batching into scheduling decisions to improve throughput and avoid the overhead of swapping models in and out of GPU memory. Our approach employs a new notion of request priority, which navigates the trade-off between attaining high accuracy and satisfying deadlines. Using data and models from three real-world applications, we show that our proposed approaches result in higher-utility schedules and higher accuracy inferences in these hardware-constrained environments.