Accelerating Error Correction Code Transformers
Matan Levy, Yoni Choukroun, Lior Wolf
TL;DR
This paper introduces a novel, resource-efficient transformer-based error correction decoder that employs ternary quantization, optimized self-attention, and graph-based positional encoding to significantly reduce computational demands while maintaining high performance.
Contribution
It presents a new acceleration method for transformer decoders in error correction, including ternary weight quantization, code-aware multi-heads, and Tanner graph eigendecomposition.
Findings
Achieves 90% model compression
Reduces energy consumption by at least 224 times
Maintains or surpasses original ECCT performance
Abstract
Error correction codes (ECC) are crucial for ensuring reliable information transmission in communication systems. Choukroun & Wolf (2022b) recently introduced the Error Correction Code Transformer (ECCT), which has demonstrated promising performance across various transmission channels and families of codes. However, its high computational and memory demands limit its practical applications compared to traditional decoding algorithms. Achieving effective quantization of the ECCT presents significant challenges due to its inherently small architecture, since existing, very low-precision quantization techniques often lead to performance degradation in compact neural networks. In this paper, we introduce a novel acceleration method for transformer-based decoders. We first propose a ternary weight quantization method specifically designed for the ECCT, inducing a decoder with…
Peer Reviews
Decision·Submitted to ICLR 2025
1. Tackles an important problem of reducing the complexity of neural decoders. 2. The performance improvements in terms of memeory and compute are non-trivial and very impressive. 3. All the three main ideas proposed are novel and interesting. Specifically, splitting the masking based on the node structure is a clever example of leveraging domain structure. 4. Set of experiements are exhaustive and includes sufficient number of results as well as ablation studies to show the merits of prop
1. While a lot of interesting ideas were discussed in the paper to improve the efficiency of ECCT, I am not fully convinced about the practical relevance of a "universal decoder" architecture that simply utilizes a parity check matrix for all codes. In coding theory, each well-known family of codes has highly specific representations and special propoerties that cannot be fully captured by a simple parity check matrix. For instance, in Polar codes, the reliability sequence and the sequential dec
1. The paper is well written and easy to follow, and the contributions are relevant. 2. The techniques introduced demonstrate significant compression without performance degradation compared to ECCTs. 3. The technique is tested on 3 different ECC types - Polar, Low-Density Parity Check (LDPC), and Bose–Chaudhuri–Hocquenghem (BCH) codes - demonstrating robustness and applicability across various error correction scenarios. 4. The authors conduct ablation studies evaluating the impact of each of t
1. Two out of three proposed techniques (HPSA and SPE) may have limited applicability outside of ECC. 2. It is not clear how easy it will be to implement these techniques in existing hardware. Ternary quantization has been explored by previous work. Beyond quantization, does HPSA and SPE introduce additional implementation complexities on existing hardware? 3. The energy efficiency numbers are estimates, the paper does not provide any real measured numbers on actual hardware.
This paper presents contributions to enhancing ECCT's practical applicability through a comprehensive optimization approach. The key strengths lie in its multi-faceted architectural improvements and efficient implementation strategies. 1. The paper introduces three well-designed technical innovations that directly address ECCT's structural limitations. The Head Partitioning Self Attention (HPSA) mechanism optimizes the self-attention computation specifically for bipartite graph message passing,
The weaknesses of this work are manifested in the following three aspects: 1. Limited Scope and Generalization. The method's exclusive focus on ECCT architecture raises concerns about its broader impact. As ECCT has not gained significant attention in academia nor found applications in industry, optimizations specific to this architecture may have limited practical value. 2. Lack of Technical Novelty in Quantization. The proposed AAP quantization method, while showing good performance, primari
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRadiation Effects in Electronics · VLSI and Analog Circuit Testing · Software Testing and Debugging Techniques
MethodsDense Connections · Adam · Linear Layer · Residual Connection · Position-Wise Feed-Forward Layer · Attention Is All You Need · Label Smoothing · Dropout · Byte Pair Encoding · Absolute Position Encodings