Performance and Complexity Analysis of bi-directional Recurrent Neural Network Models vs. Volterra Nonlinear Equalizers in Digital Coherent Systems

TL;DR

This paper compares bi-directional RNN models and Volterra nonlinear equalizers for fiber nonlinearity compensation in digital coherent optical systems, highlighting the efficiency and performance of RNNs, especially the Vanilla-RNN, in real-world applications.

Contribution

It demonstrates that bi-directional RNNs, particularly the Vanilla-RNN, outperform Volterra equalizers in complexity and performance for fiber nonlinearity mitigation.

Findings

All RNN models perform similarly in compensation.

Vanilla-RNN is preferred due to simplicity and comparable performance.

RNN processing surpasses Volterra equalizers in efficiency.

Abstract

We investigate the complexity and performance of recurrent neural network (RNN) models as post-processing units for the compensation of fibre nonlinearities in digital coherent systems carrying polarization multiplexed 16-QAM and 32-QAM signals. We evaluate three bi-directional RNN models, namely the bi-LSTM, bi-GRU and bi-Vanilla-RNN and show that all of them are promising nonlinearity compensators especially in dispersion unmanaged systems. Our simulations show that during inference the three models provide similar compensation performance, therefore in real-life systems the simplest scheme based on Vanilla-RNN units should be preferred. We compare bi-Vanilla-RNN with Volterra nonlinear equalizers and exhibit its superiority both in terms of performance and complexity, thus highlighting that RNN processing is a very promising pathway for the upgrade of long-haul optical communication systems utilizing coherent detection.