SoftNeuro: Fast Deep Inference using Multi-platform Optimization

TL;DR

SoftNeuro is a high-performance deep learning inference framework that optimizes speed on various platforms by profiling routines and selecting the fastest execution paths, significantly improving inference efficiency.

Contribution

The paper introduces SoftNeuro, a novel framework that separates routines from network layers and uses dynamic programming for optimal routine selection, enhancing inference speed and tuning efficiency.

Findings

Achieves faster inference on edge and server devices.

Demonstrates effective routine profiling and selection.

Outperforms existing inference methods in speed and tuning efficiency.

Abstract

Faster inference of deep learning models is highly demanded on edge devices and even servers, for both financial and environmental reasons. To address this issue, we propose SoftNeuro, a novel, high-performance inference framework with efficient performance tuning. The key idea is to separate algorithmic routines from network layers. Our framework maximizes the inference performance by profiling various routines for each layer and selecting the fastest path. To efficiently find the best path, we propose a routine-selection algorithm based on dynamic programming. Experiments show that the proposed framework achieves both fast inference and efficient tuning.