Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

TL;DR

Woodpecker-DL is a hardware-aware framework that optimizes deep neural network inference through graph optimization, automated searches, DSL compilation, and system-level exploration, achieving significant speedups on GPU hardware.

Contribution

It introduces a novel hardware-aware optimization framework with automated search methods and a DSL compiler for efficient inference acceleration.

Findings

Achieved up to 5.40x speedup over cuDNN on convolution operators.

Ran up to 1.18x faster than TensorRT for end-to-end inference.

Demonstrated effectiveness on Tesla P100 GPU with multiple optimization techniques.

Abstract

Accelerating deep model training and inference is crucial in practice. Existing deep learning frameworks usually concentrate on optimizing training speed and pay fewer attentions to inference-specific optimizations. Actually, model inference differs from training in terms of computation, e.g. parameters are refreshed each gradient update step during training, but kept invariant during inference. These special characteristics of model inference open new opportunities for its optimization. In this paper, we propose a hardware-aware optimization framework, namely Woodpecker-DL (WPK), to accelerate inference by taking advantage of multiple joint optimizations from the perspectives of graph optimization, automated searches, domain-specific language (DSL) compiler techniques and system-level exploration. In WPK, we investigated two new automated search approaches based on genetic algorithm and reinforcement learning, respectively, to hunt the best operator code configurations targeting specific hardware. A customized DSL compiler is further attached to these search algorithms to generate efficient codes. To create an optimized inference plan, WPK systematically explores high-speed operator implementations from third-party libraries besides our automatically generated codes and singles out the best implementation per operator for use. Extensive experiments demonstrated that on a Tesla P100 GPU, we can achieve the maximum speedup of 5.40 over cuDNN and 1.63 over TVM on individual convolution operators, and run up to 1.18 times faster than TensorRT for end-to-end model inference.