Quantization of Neural Network Equalizers in Optical Fiber Transmission Experiments

TL;DR

This paper demonstrates that low-complexity neural network equalizers, with quantized weights and activations, can effectively mitigate nonlinearities in optical fiber transmission, significantly reducing computational complexity while maintaining performance.

Contribution

It introduces novel quantization algorithms, including companding successive alpha-blending, for neural network equalizers in optical fiber communication, enabling low-bit quantization with minimal performance loss.

Findings

Quantization-aware training with straight-through estimation incurs less than 0.5 dB penalty at ≥5 bits.

Companding successive alpha-blending effectively compensates for quantization errors in low-bit regimes.

Neural network complexity is reduced by over 90% with minimal performance impact.

Abstract

The quantization of neural networks for the mitigation of the nonlinear and components' distortions in dual-polarization optical fiber transmission is studied. Two low-complexity neural network equalizers are applied in three 16-QAM 34.4 GBaud transmission experiments with different representative fibers. A number of post-training quantization and quantization-aware training algorithms are compared for casting the weights and activations of the neural network in few bits, combined with the uniform, additive power-of-two, and companding quantization. For quantization in the large bit-width regime of $\geq 5$ bits, the quantization-aware training with the straight-through estimation incurs a Q-factor penalty of less than 0.5 dB compared to the unquantized neural network. For quantization in the low bit-width regime, an algorithm dubbed companding successive alpha-blending quantization is suggested. This method compensates for the quantization error aggressively by successive grouping and retraining of the parameters, as well as an incremental transition from the floating-point representations to the quantized values within each group. The activations can be quantized at 8 bits and the weights on average at 1.75 bits, with a penalty of $\leq 0.5$~dB. If the activations are quantized at 6 bits, the weights can be quantized at 3.75 bits with minimal penalty. The computational complexity and required storage of the neural networks are drastically reduced, typically by over 90\%. The results indicate that low-complexity neural networks can mitigate nonlinearities in optical fiber transmission.