Dynamic Network Adaptation at Inference

TL;DR

This paper introduces SLO-Aware Neural Networks that dynamically adjust computation during inference by dropping nodes, enabling real-time systems to meet diverse latency and accuracy requirements efficiently.

Contribution

It proposes a novel method for per-inference dynamic node dropout based on SLO targets, improving speed and maintaining accuracy in inference-serving systems.

Findings

Achieves 1.3-56.7× speedups with minimal accuracy loss

Serves multiple accuracy targets with a single trained model

Mitigates latency degradation from co-location interference

Abstract

Machine learning (ML) inference is a real-time workload that must comply with strict Service Level Objectives (SLOs), including latency and accuracy targets. Unfortunately, ensuring that SLOs are not violated in inference-serving systems is challenging due to inherent model accuracy-latency tradeoffs, SLO diversity across and within application domains, evolution of SLOs over time, unpredictable query patterns, and co-location interference. In this paper, we observe that neural networks exhibit high degrees of per-input activation sparsity during inference. . Thus, we propose SLO-Aware Neural Networks which dynamically drop out nodes per-inference query, thereby tuning the amount of computation performed, according to specified SLO optimization targets and machine utilization. SLO-Aware Neural Networks achieve average speedups of $1.3-56.7\times$ with little to no accuracy loss (less than 0.3%). When accuracy constrained, SLO-Aware Neural Networks are able to serve a range of accuracy targets at low latency with the same trained model. When latency constrained, SLO-Aware Neural Networks can proactively alleviate latency degradation from co-location interference while maintaining high accuracy to meet latency constraints.