Arch-Net: Model Distillation for Architecture Agnostic Model Deployment

TL;DR

Arch-Net introduces a model distillation approach to convert complex neural networks into architecture-agnostic, efficient models suitable for deployment across various ASIC chips, addressing hardware support limitations.

Contribution

The paper presents a novel label-free blockwise model distillation method combined with quantization to produce architecture-agnostic neural networks for ASIC deployment.

Findings

Transform neural architectures into fast, accurate models for ASICs

Effective removal of non-supported constructs like Layer Norm and Embeddings

Achieves high performance on translation and image classification tasks

Abstract

Vast requirement of computation power of Deep Neural Networks is a major hurdle to their real world applications. Many recent Application Specific Integrated Circuit (ASIC) chips feature dedicated hardware support for Neural Network Acceleration. However, as ASICs take multiple years to develop, they are inevitably out-paced by the latest development in Neural Architecture Research. For example, Transformer Networks do not have native support on many popular chips, and hence are difficult to deploy. In this paper, we propose Arch-Net, a family of Neural Networks made up of only operators efficiently supported across most architectures of ASICs. When a Arch-Net is produced, less common network constructs, like Layer Normalization and Embedding Layers, are eliminated in a progressive manner through label-free Blockwise Model Distillation, while performing sub-eight bit quantization at the same time to maximize performance. Empirical results on machine translation and image classification tasks confirm that we can transform latest developed Neural Architectures into fast running and as-accurate Arch-Net, ready for deployment on multiple mass-produced ASIC chips. The code will be available at https://github.com/megvii-research/Arch-Net.