Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets

TL;DR

This paper introduces a two-stage sub 8-bit quantization aware training method for streaming keyword spotting models, achieving near-floating point accuracy while significantly reducing compute and memory requirements on embedded chipsets.

Contribution

The paper presents a novel 2-stage quantization algorithm for all model components, enabling efficient deployment of keyword spotting models on embedded hardware with minimal accuracy loss.

Findings

Achieves parity with full floating point model on detection error tradeoff curve.

Up to 3x CPU efficiency improvement and 4x memory reduction.

Effective quantization across various bit widths (1, 4, 5, 8-bit).

Abstract

We propose a novel 2-stage sub 8-bit quantization aware training algorithm for all components of a 250K parameter feedforward, streaming, state-free keyword spotting model. For the 1st-stage, we adapt a recently proposed quantization technique using a non-linear transformation with tanh(.) on dense layer weights. In the 2nd-stage, we use linear quantization methods on the rest of the network, including other parameters (bias, gain, batchnorm), inputs, and activations. We conduct large scale experiments, training on 26,000 hours of de-identified production, far-field and near-field audio data (evaluating on 4,000 hours of data). We organize our results in two embedded chipset settings: a) with commodity ARM NEON instruction set and 8-bit containers, we present accuracy, CPU, and memory results using sub 8-bit weights (4, 5, 8-bit) and 8-bit quantization of rest of the network; b) with off-the-shelf neural network accelerators, for a range of weight bit widths (1 and 5-bit), while presenting accuracy results, we project reduction in memory utilization. In both configurations, our results show that the proposed algorithm can achieve: a) parity with a full floating point model's operating point on a detection error tradeoff (DET) curve in terms of false detection rate (FDR) at false rejection rate (FRR); b) significant reduction in compute and memory, yielding up to 3 times improvement in CPU consumption and more than 4 times improvement in memory consumption.