An efficient and flexible inference system for serving heterogeneous ensembles of deep neural networks

TL;DR

This paper introduces a novel inference system for serving ensembles of deep neural networks efficiently and flexibly across heterogeneous hardware, optimizing resource allocation and asynchronous processing.

Contribution

It presents a new software layer with innovative allocation and asynchronous execution techniques for ensemble DNN inference, outperforming existing methods.

Findings

Successfully serves 12 heavy DNNs on 4 GPUs

Achieves up to 2.7x speedup over baseline

Supports multi-GPU multi-threaded DNN inference

Abstract

Ensembles of Deep Neural Networks (DNNs) have achieved qualitative predictions but they are computing and memory intensive. Therefore, the demand is growing to make them answer a heavy workload of requests with available computational resources. Unlike recent initiatives on inference servers and inference frameworks, which focus on the prediction of single DNNs, we propose a new software layer to serve with flexibility and efficiency ensembles of DNNs. Our inference system is designed with several technical innovations. First, we propose a novel procedure to find a good allocation matrix between devices (CPUs or GPUs) and DNN instances. It runs successively a worst-fit to allocate DNNs into the memory devices and a greedy algorithm to optimize allocation settings and speed up the ensemble. Second, we design the inference system based on multiple processes to run asynchronously: batching, prediction, and the combination rule with an efficient internal communication scheme to avoid overhead. Experiments show the flexibility and efficiency under extreme scenarios: It successes to serve an ensemble of 12 heavy DNNs into 4 GPUs and at the opposite, one single DNN multi-threaded into 16 GPUs. It also outperforms the simple baseline consisting of optimizing the batch size of DNNs by a speedup up to 2.7X on the image classification task.