BiFSMN: Binary Neural Network for Keyword Spotting

TL;DR

BiFSMN is a highly efficient binary neural network designed for keyword spotting on edge devices, combining novel training schemes and architecture optimizations to achieve high accuracy with significant speed and storage improvements.

Contribution

The paper introduces BiFSMN, a binary neural network with a high-frequency enhancement distillation scheme and a thinnable architecture, enabling efficient deployment for keyword spotting.

Findings

Outperforms existing binarization methods with less than 3% accuracy drop.

Achieves 22.3x speedup and 15.5x storage reduction on edge hardware.

Comparable accuracy to full-precision models on Speech Commands dataset.

Abstract

The deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. However, computational resources for these networks are significantly constrained since they usually run on-call on edge devices. In this paper, we present BiFSMN, an accurate and extreme-efficient binary neural network for KWS. We first construct a High-frequency Enhancement Distillation scheme for the binarization-aware training, which emphasizes the high-frequency information from the full-precision network's representation that is more crucial for the optimization of the binarized network. Then, to allow the instant and adaptive accuracy-efficiency trade-offs at runtime, we also propose a Thinnable Binarization Architecture to further liberate the acceleration potential of the binarized network from the topology perspective. Moreover, we implement a Fast Bitwise Computation Kernel for BiFSMN on ARMv8 devices which fully utilizes registers and increases instruction throughput to push the limit of deployment efficiency. Extensive experiments show that BiFSMN outperforms existing binarization methods by convincing margins on various datasets and is even comparable with the full-precision counterpart (e.g., less than 3% drop on Speech Commands V1-12). We highlight that benefiting from the thinnable architecture and the optimized 1-bit implementation, BiFSMN can achieve an impressive 22.3x speedup and 15.5x storage-saving on real-world edge hardware. Our code is released at https://github.com/htqin/BiFSMN.