ALT: Boosting Deep Learning Performance by Breaking the Wall between Graph and Operator Level Optimizations

TL;DR

ALT is a deep learning compiler that unifies graph and operator-level optimizations, leading to significant performance improvements in inference speed on heterogeneous hardware.

Contribution

It introduces a joint optimization framework combining graph and operator tuning, which was previously separated in existing compilers.

Findings

Achieves 1.5x speedup on single operators

Attains 1.4x faster end-to-end inference

Outperforms state-of-the-art compilers like Ansor

Abstract

Deep learning models rely on highly optimized tensor libraries for efficient inference on heterogeneous hardware. Current deep compilers typically predetermine layouts of tensors and then optimize loops of operators. However, such unidirectional and one-off workflow strictly separates graph-level optimization and operator-level optimization into different system layers, missing opportunities for unified tuning. This paper proposes ALT, a compiler that performs joint graph- and operator-level optimizations for deep models. ALT provides a generic transformation module to manipulate layouts and loops with easy-to-use primitive functions. ALT further integrates an auto-tuning module that jointly optimizes graph-level data layouts and operator-level loops while guaranteeing efficiency. Experimental results show that ALT significantly outperforms state-of-the-art compilers (e.g., Ansor) in terms of both single operator performance (e.g., 1.5x speedup on average) and end-to-end inference performance (e.g., 1.4x speedup on average).