Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation

TL;DR

This paper introduces a low-complexity neural network equalizer for optical systems, using advanced model compression techniques to enhance performance while reducing implementation complexity.

Contribution

It presents a comprehensive comparison of deep model compression methods and proposes a Bayesian optimization-assisted approach for neural network equalizer compression.

Findings

Optimal compression improves equalizer performance and reduces multipliers.

Compressed neural equalizers outperform conventional digital back-propagation.

Equalizers achieve similar complexity to chromatic dispersion compensation with better results.

Abstract

In this paper, a new methodology is proposed that allows for the low-complexity development of neural network (NN) based equalizers for the mitigation of impairments in high-speed coherent optical transmission systems. In this work, we provide a comprehensive description and comparison of various deep model compression approaches that have been applied to feed-forward and recurrent NN designs. Additionally, we evaluate the influence these strategies have on the performance of each NN equalizer. Quantization, weight clustering, pruning, and other cutting-edge strategies for model compression are taken into consideration. In this work, we propose and evaluate a Bayesian optimization-assisted compression, in which the hyperparameters of the compression are chosen to simultaneously reduce complexity and improve performance. In conclusion, the trade-off between the complexity of each compression approach and its performance is evaluated by utilizing both simulated and experimental data in order to complete the analysis. By utilizing optimal compression approaches, we show that it is possible to design an NN-based equalizer that is simpler to implement and has better performance than the conventional digital back-propagation (DBP) equalizer with only one step per span. This is accomplished by reducing the number of multipliers used in the NN equalizer after applying the weighted clustering and pruning algorithms. Furthermore, we demonstrate that an equalizer based on NN can also achieve superior performance while still maintaining the same degree of complexity as the full electronic chromatic dispersion compensation block. We conclude our analysis by highlighting open questions and existing challenges, as well as possible future research directions.