A Novel CNN Based Standalone Detector for Faster-than-Nyquist Signaling

TL;DR

This paper introduces a CNN-based detector for faster-than-Nyquist signaling that uses fixed kernels and hierarchical filters to effectively mitigate intersymbol interference, achieving near-optimal performance with reduced computational cost.

Contribution

The proposed detector employs structured fixed kernel layers with domain-informed masking and a hierarchical filter strategy, offering improved accuracy and efficiency over traditional methods.

Findings

Achieves near-optimal BER performance comparable to BCJR algorithm for $ au \,\geq\, 0.7$

Reduces computational cost by up to 46% for BPSK and 84% for QPSK

Demonstrates robustness in high-order modulations and fading channels

Abstract

This paper presents a novel convolutional neural network (CNN)-based detector for faster-than-Nyquist (FTN) signaling, introducing structured fixed kernel layers with domain-informed masking to effectively mitigate intersymbol interference (ISI). Unlike standard CNN architectures that rely on moving kernels, the proposed approach employs fixed convolutional kernels at predefined positions to explicitly learn ISI patterns at varying distances from the central symbol. To enhance feature extraction, a hierarchical filter allocation strategy is employed, assigning more filters to earlier layers for stronger ISI components and fewer to later layers for weaker components. This structured design improves feature representation, eliminates redundant computations, and enhances detection accuracy while maintaining computational efficiency. Simulation results demonstrate that the proposed detector achieves near-optimal bit error rate (BER) performance, comparable to the BCJR algorithm for the compression factor $\tau \geq 0.7$, while offering up to $46\%$ and $84\%$ computational cost reduction over M-BCJR for BPSK and QPSK, respectively. Additional evaluations confirm the method's adaptability to high-order modulations (up to 64-QAM), resilience in quasi-static multipath Rayleigh fading channels, and effectiveness under LDPC-coded FTN transmission, highlighting its robustness and practicality.