A Novel Domain-Aware CNN Architecture for Faster-than-Nyquist Signaling Detection

TL;DR

This paper introduces a fixed-kernel CNN architecture with domain-informed masking for faster-than-Nyquist signaling detection, achieving near-optimal BER with significant computational efficiency improvements.

Contribution

It presents the first fixed-kernel CNN design tailored for FTN detection, incorporating domain-aware masking and hierarchical filter allocation for improved accuracy and efficiency.

Findings

Achieves near-BCJR BER performance for τ ≥ 0.7

Reduces computational complexity by up to 46% and 84% for BPSK and QPSK

Demonstrates superior efficiency over existing methods

Abstract

This paper proposes a convolutional neural network (CNN)-based detector for faster-than-Nyquist (FTN) signaling that employs structured fixed kernel layers with domain-informed masking to mitigate intersymbol interference (ISI). Unlike standard CNNs with sliding kernels, the proposed method utilizes fixed-position kernels to directly capture ISI effects at varying distances from the central symbol. A hierarchical filter allocation strategy is also introduced, assigning more filters to earlier layers for strong ISI patterns and fewer to later layers for weaker ones. This design improves detection accuracy while reducing redundant operations. Simulation results show that the detector achieves near-optimal bit error rate (BER) performance for $\tau \geq 0.7$, closely matching the BCJR algorithm, and offers computational gains of up to $46\%$ and $84\%$ over M-BCJR for BPSK and QPSK, respectively. Comparative analysis with other methods further highlights the efficiency and effectiveness of the proposed approach. To the best of our knowledge, this is the first application of a fixed-kernel CNN architecture tailored for FTN detection in the literature.