Deep Learning Inference Frameworks Benchmark

TL;DR

This paper provides an empirical comparison of four deep learning inference frameworks, analyzing their performance across various configurations and exploring optimization opportunities for co-localized models on GPUs.

Contribution

It offers the first detailed analysis of how different configurations affect inference performance and identifies new opportunities for accelerating ensemble models on GPUs.

Findings

Different configurations significantly impact speed, memory, and computation.

No single framework dominates in performance.

Opportunities exist for optimizing ensemble models on the same GPU.

Abstract

Deep learning (DL) has been widely adopted those last years but they are computing-intensive method. Therefore, scientists proposed diverse optimization to accelerate their predictions for end-user applications. However, no single inference framework currently dominates in terms of performance. This paper takes a holistic approach to conduct an empirical comparison and analysis of four representative DL inference frameworks. First, given a selection of CPU-GPU configurations, we show that for a specific DL framework, different configurations of its settings may have a significant impact on the prediction speed, memory, and computing power. Second, to the best of our knowledge, this study is the first to identify the opportunities for accelerating the ensemble of co-localized models in the same GPU. This measurement study provides an in-depth empirical comparison and analysis of four representative DL frameworks and offers practical guidance for service providers to deploy and deliver DL predictions.