ML-Enabled Deformable Matched Filters for Bandlimitation Compensation in Free-Space Optics

TL;DR

This paper introduces a neural-network-assisted deformable matched filter for CAP modulation in bandwidth-limited free-space optics, enabling adaptive pulse-shape compensation and improved performance over traditional fixed filters.

Contribution

It presents a novel neural network-based deformable matched filtering framework that learns residual deformations of the nominal filter using physically motivated features.

Findings

Significantly outperforms conventional fixed matched filters under severe bandwidth constraints.

Does not require decision feedback or increased receiver latency.

Demonstrated effectiveness in a hardware-in-the-loop CAP transmission system.

Abstract

This paper proposes a neural-network-assisted deformable matched filtering framework for carrier-less amplitude and phase (CAP) modulation operating under bandwidth-limited channel conditions. Instead of replacing the analytically derived CAP matched filter, the proposed receiver learns a residual deformation of the nominal matched filter based on a compact set of physically motivated signal features extracted from the received waveform. A total of 16 time-domain, frequency-domain, and memory-related features are used to provide a low-dimensional representation of bandwidth-induced pulse distortion. These features are mapped by a fully connected neural network to complex-valued matched filter coefficients, enabling adaptive pulse-shape compensation prior to symbol-rate sampling. The network is trained end-to-end using a differentiable loss function based on error vector magnitude (EVM). Experimental results obtained using a hardware-in-the-loop CAP transmission system demonstrate that the proposed deformable matched filter significantly outperforms conventional fixed matched filtering under severe bandwidth constraints, without requiring decision feedback or increasing receiver latency.