Hybrid Mamba-Transformer Decoder for Error-Correcting Codes

TL;DR

This paper presents a hybrid deep learning decoder combining Mamba architecture and Transformer layers, utilizing a novel masking strategy and progressive loss to improve error correction performance.

Contribution

It introduces a new hybrid Mamba-Transformer decoder with layer-wise masking and progressive supervision, advancing decoding accuracy for error-correcting codes.

Findings

Outperforms Transformer-only decoders

Surpasses standard Mamba models in accuracy

Effective across various linear codes

Abstract

We introduce a novel deep learning method for decoding error correction codes based on the Mamba architecture, enhanced with Transformer layers. Our approach proposes a hybrid decoder that leverages Mamba's efficient sequential modeling while maintaining the global context capabilities of Transformers. To further improve performance, we design a novel layer-wise masking strategy applied to each Mamba layer, allowing selective attention to relevant code features at different depths. Additionally, we introduce a progressive layer-wise loss, supervising the network at intermediate stages and promoting robust feature extraction throughout the decoding process. Comprehensive experiments across a range of linear codes demonstrate that our method significantly outperforms Transformer-only decoders and standard Mamba models.