Throughput Maximization of DNN Inference: Batching or Multi-Tenancy?

TL;DR

This paper introduces DNNScaler, a system that dynamically chooses between batching and multi-tenancy to maximize DNN inference throughput on GPUs while satisfying latency constraints, demonstrating up to 14x improvement.

Contribution

DNNScaler automatically detects the optimal approach for DNN inference throughput and adjusts parameters to meet latency, outperforming previous methods significantly.

Findings

Up to 14x throughput improvement over prior approaches

DNNScaler effectively maintains latency while increasing throughput

Performance gains are consistent across various DNN architectures and datasets

Abstract

Deployment of real-time ML services on warehouse-scale infrastructures is on the increase. Therefore, decreasing latency and increasing throughput of deep neural network (DNN) inference applications that empower those services have attracted attention from both academia and industry. A common solution to address this challenge is leveraging hardware accelerators such as GPUs. To improve the inference throughput of DNNs deployed on GPU accelerators, two common approaches are employed: Batching and Multi-Tenancy. Our preliminary experiments show that the effect of these approaches on the throughput depends on the DNN architecture. Taking this observation into account, we design and implement DNNScaler which aims to maximize the throughput of interactive AI-powered services while meeting their latency requirements. DNNScaler first detects the suitable approach (Batching or Multi-Tenancy) that would be most beneficial for a DNN regarding throughput improvement. Then, it adjusts the control knob of the detected approach (batch size for Batching and number of co-located instances for Multi-Tenancy) to maintain the latency while increasing the throughput. Conducting an extensive set of experiments using well-known DNNs from a variety of domains, several popular datasets, and a cutting-edge GPU, the results indicate that DNNScaler can improve the throughput by up to 14x (218% on average) compared with the previously proposed approach, while meeting the latency requirements of the services.