Transformer-based Nonlinear Equalization for DP-16QAM Coherent Optical Communication Systems

TL;DR

This paper introduces a Transformer-based neural network model to mitigate nonlinear effects in DP-16QAM optical communication systems, demonstrating improved performance over existing methods like DBP, FCNN, and BiLSTM.

Contribution

The paper presents the first application of Transformer architecture for nonlinear equalization in coherent optical systems, outperforming traditional and neural network-based approaches.

Findings

Transformer model outperforms DBP, FCNN, and BiLSTM in nonlinear compensation.

Performance gains are consistent across different fiber lengths and power levels.

The approach reduces computational complexity compared to traditional methods.

Abstract

Compensating for nonlinear effects using digital signal processing (DSP) is complex and computationally expensive in long-haul optical communication systems due to intractable interactions between Kerr nonlinearity, chromatic dispersion (CD), and amplified spontaneous emission (ASE) noise from inline amplifiers. The application of machine learning architectures has demonstrated promising advancements in enhancing transmission performance through the mitigation of fiber nonlinear effects. In this paper, we apply a Transformer-based model to dual-polarisation (DP)-16QAM coherent optical communication systems. We test the performance of the proposed model for different values of fiber lengths and launched optical powers and show improved performance compared to the state-of-the-art digital backpropagation (DBP) algorithm, fully connected neural network (FCNN) and bidirectional long short term memory (BiLSTM) architecture.