FusionStitching: Deep Fusion and Code Generation for Tensorflow Computations on GPUs

TL;DR

FusionStitching is a new system that fuses TensorFlow operations into large GPU kernels, significantly reducing kernel launches and boosting overall performance for machine learning workloads.

Contribution

It introduces a comprehensive fusion and code generation approach that improves GPU utilization for TensorFlow computations beyond existing XLA methods.

Findings

55% reduction in GPU kernel launches

Up to 20% increase in end-to-end application performance

Effective on multiple public and in-house models

Abstract

In recent years, there is a surge on machine learning applications in industry. Many of them are based on popular AI frameworks like Tensorflow, Torch, Caffe, or MxNet, etc, and are enpowered by accelerator platforms such as GPUs. One important challenge of running Tensorflow computations on GPUs is the fine granularity problem, namely, FLOPS of individual ops are far from enough to fully exploit the computing power of underlying accelerators. The XLA framework provides a solid foundation to explore this problem further. In this paper, we propose FusionStitching, a novel, comprehensive Op fusion and code generation system to stitch computations into large GPU kernels. Experimental results on four public models and two of our large inhouse applications show another 55% (geometric mean) reduction of GPU kernel launches, compared to the XLA fusion baseline. This increases the E2E performance of both of our latency critical inhouse applications up to 20%.