FTuner: A Fast Dynamic Shape Tensors Program Auto-Tuner for Deep Learning Compilers

TL;DR

FTuner is a novel auto-tuning technique for deep learning compilers that efficiently matches dynamic tensor shapes using an abstract computational unit, significantly reducing tuning time while maintaining high performance.

Contribution

Introduces FTuner, a new method that uses the uKernel concept and hardware modeling to quickly optimize dynamic tensor shapes without extensive training or large design spaces.

Findings

Achieves 3% speedup over existing auto-tuners.

Reduces tuning time by two orders of magnitude.

Matches performance of vendor libraries.

Abstract

Many artificial intelligence models process input data of different lengths and resolutions, making the shape of the tensors dynamic. The performance of these models depends on the shape of the tensors, which makes it difficult to optimize the tensors before the model runs. There are two common solutions to this problem. The first is to add useless data to the input to match a pre-optimized tensor library. The second is to use small basic tensors to create a tensor that is closest in size to the input data and then tune it to minimize padding. However, this second solution can be time-consuming. This paper proposes a new technique for deep learning compilers called FTuner. Instead of using a large design space or training a cost model, we use an abstract computational unit called the uKernel to patch together small, various-sized tensors to match the shape of the input tensor. We determine the shape of the uKernel using an analytic hardware information model. Experiments show that the FTuner can achieve comparable operators and end-to-end performance to vendor libraries and achieves 3\% speedup on existing auto-tuner with the model-training compiler while reducing tuning time by two orders of magnitude.