# Quantization Loss Re-Learning Method

## TL;DR

This paper introduces a novel quantization loss re-learning method for LSTM gate parameters that minimizes recognition performance degradation by allowing weight parameters to compensate for quantization loss during training.

## Contribution

The paper proposes a new quantization technique that enables near-lossless quantization of LSTM gate parameters through a re-learning process during training.

## Key findings

- Gate parameters quantized to 0, 0.5, 1 with minimal F1 score decrease
- Method theoretically validated and experimentally effective
- Only 0.7% F1 score drop on NER dataset

## Abstract

In order to quantize the gate parameters of the LSTM (Long Short-Term Memory) neural network model with almost no recognition performance degraded, a new quantization method named Quantization Loss Re-Learn Method is proposed in this paper. The method does lossy quantization on gate parameters during training iterations, and the weight parameters learn to offset the loss of gate parameters quantization by adjusting the gradient in back propagation during weight parameters optimization. We proved the effectiveness of this method through theoretical derivation and experiments. The gate parameters had been quantized to 0, 0.5, 1 three values, and on the Named Entity Recognition dataset, the F1 score of the model with the new quantization method on gate parameters decreased by only 0.7% compared to the baseline model.