Blind Equalization and Channel Estimation in Coherent Optical Communications Using Variational Autoencoders

TL;DR

This paper explores variational autoencoder-based blind equalizers for optical communication channels, demonstrating improved performance over traditional algorithms in complex, high-order modulation scenarios with channel variability.

Contribution

It introduces a novel VAE-based equalizer framework that generalizes to higher order modulation, dual-polarization, and probabilistic shaping, with both model-based and black-box approaches.

Findings

VAE equalizer outperforms CMA in PCS scenarios

Effective channel estimation via VAE in dispersive channels

Enhanced adaptability to time-varying channels

Abstract

We investigate the potential of adaptive blind equalizers based on variational inference for carrier recovery in optical communications. These equalizers are based on a low-complexity approximation of maximum likelihood channel estimation. We generalize the concept of variational autoencoder (VAE) equalizers to higher order modulation formats encompassing probabilistic constellation shaping (PCS), ubiquitous in optical communications, oversampling at the receiver, and dual-polarization transmission. Besides black-box equalizers based on convolutional neural networks, we propose a model-based equalizer based on a linear butterfly filter and train the filter coefficients using the variational inference paradigm. As a byproduct, the VAE also provides a reliable channel estimation. We analyze the VAE in terms of performance and flexibility over a classical additive white Gaussian noise (AWGN) channel with inter-symbol interference (ISI) and over a dispersive linear optical dual-polarization channel. We show that it can extend the application range of blind adaptive equalizers by outperforming the state-of-the-art constant-modulus algorithm (CMA) for PCS for both fixed but also time-varying channels. The evaluation is accompanied with a hyperparameter analysis.