Canvas: End-to-End Kernel Architecture Search in Neural Networks

TL;DR

Canvas introduces Kernel Architecture Search (KAS), an end-to-end framework that automatically generates high-performance neural kernels, leading to significant speedups in neural networks with minimal accuracy loss.

Contribution

The paper presents a novel system, Canvas, that combines NAS and tensor compilation into Kernel Architecture Search, enabling automated, fine-grained kernel design for improved neural network performance.

Findings

Achieves 1.5x speedup over state-of-the-art methods.

Successfully rediscovered manually designed kernels.

Generates new kernels that inspire future ML innovations.

Abstract

The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations. For source code and implementation, we open-sourced Canvas at https://github.com/tsinghua-ideal/Canvas.