A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators

TL;DR

This paper introduces FAST, a comprehensive search framework for designing domain-optimized deep learning accelerators that significantly improve performance and efficiency for specific workloads in datacenter environments.

Contribution

The paper presents FAST, a full-stack search framework that optimizes hardware and software design decisions for deep learning accelerators tailored to specific workloads.

Findings

FAST accelerators improve Perf/TDP by 3.7x on average for single workloads.

FAST accelerators improve Perf/TDP by 2.4x on average for multi-workload serving.

FAST-generated accelerators are practical for moderate-sized datacenter deployments.

Abstract

The rapidly-changing deep learning landscape presents a unique opportunity for building inference accelerators optimized for specific datacenter-scale workloads. We propose Full-stack Accelerator Search Technique (FAST), a hardware accelerator search framework that defines a broad optimization environment covering key design decisions within the hardware-software stack, including hardware datapath, software scheduling, and compiler passes such as operation fusion and tensor padding. In this paper, we analyze bottlenecks in state-of-the-art vision and natural language processing (NLP) models, including EfficientNet and BERT, and use FAST to design accelerators capable of addressing these bottlenecks. FAST-generated accelerators optimized for single workloads improve Perf/TDP by 3.7x on average across all benchmarks compared to TPU-v3. A FAST-generated accelerator optimized for serving a suite of workloads improves Perf/TDP by 2.4x on average compared to TPU-v3. Our return on investment analysis shows that FAST-generated accelerators can potentially be practical for moderate-sized datacenter deployments.