Vortex: Efficient Sample-Free Dynamic Tensor Program Optimization via Hardware-aware Strategy Space Hierarchization

TL;DR

Vortex is a hardware-aware, sample-free compiler for dynamic-shape tensor programs that significantly improves compilation efficiency and runtime performance across CPU and GPU platforms by leveraging hardware information and strategy space hierarchization.

Contribution

Vortex introduces a novel hardware-driven, sample-free compilation approach with a bidirectional workflow and strategy space hierarchization for dynamic-shape tensor programs.

Findings

Reduces compilation time by 176x compared to existing methods.

Achieves 2.53x speedup on CPU and 3.01x on GPU over vendor libraries.

Outperforms existing dynamic-shape compilers in efficiency and performance.

Abstract

Dynamic-shape deep neural networks (DNNs) are rapidly evolving, attracting attention for their ability to handle variable input sizes in real-time applications. However, existing compilation optimization methods for such networks often rely heavily on predefined samples to guide the compilation process, which restricts their adaptability and efficiency. These sample-driven methods struggle to efficiently manage the diverse and unpredictable shapes encountered in real-world scenarios, often resulting in suboptimal performance. To tackle these issues, we introduce Vortex, a hardware-driven and sample-free compiler tailored for dynamic-shape tensor programs. Vortex capitalizes on detailed hardware information and hierarchizes the strategy space to facilitate high-performance code generation without relying on runtime shape samples. It features a unique bidirectional compilation workflow, combining top-down abstraction for aligning tensor program execution with hardware hierarchies and bottom-up kernel construction to narrow the search space, enabling Vortex to achieve remarkable efficiency. Comprehensive evaluations confirm that Vortex reduces compilation time by $176\times$ compared to the existing dynamic-shape compiler. Additionally, it substantially outperforms existing vendor-provided libraries and dynamic-shape compilers on both CPU and GPU platforms, delivering speedups of $2.53\times$ and $3.01\times$, respectively.