Robust Bidirectional Associative Memory via Regularization Inspired by the Subspace Rotation Algorithm
Ci Lin, Tet Yeap, Iluju Kiringa, Biwei Zhang
TL;DR
This paper introduces a new regularization-based training method for Bidirectional Associative Memory (BAM) that significantly enhances its robustness against noise and adversarial attacks, using principles inspired by the Subspace Rotation Algorithm.
Contribution
The paper proposes the Bidirectional Subspace Rotation Algorithm (B-SRA) and new regularization strategies that improve BAM's robustness and convergence, advancing the state-of-the-art in resilient associative memory models.
Findings
B-SRA improves robustness and convergence of BAM.
Orthogonal weight matrices and gradient-pattern alignment are key to robustness.
The combined OWM and GPA (SAME) configuration offers the best resistance to attacks.
Abstract
Bidirectional Associative Memory (BAM) trained with Bidirectional Backpropagation (B-BP) often suffers from poor robustness and high sensitivity to noise and adversarial attacks. To address these issues, we propose a novel gradient-free training algorithm, the Bidirectional Subspace Rotation Algorithm (B-SRA), which significantly improves the robustness and convergence behavior of BAM. Through comprehensive experiments, we identify two key principles -- orthogonal weight matrices (OWM) and gradient-pattern alignment (GPA) -- as central to enhancing the robustness of BAM. Motivated by these findings, we introduce new regularization strategies into B-BP, resulting in models with greatly improved resistance to corruption and adversarial perturbations. We further conduct an ablation study across different training strategies to determine the most robust configuration and evaluate BAM's…
Peer Reviews
Decision·Submitted to ICLR 2026
The paper tries to improve the robustness of BAMs. This topic is significant because of BAM's suitability for modular neuromorphic hardware design and robust learning. These and other potential benefits of BAMs have led to an increase in interest within the AI research community.
1) Unsupported claim about Bidirectional Backpropagation (B-BP): The paper makes the strong claim, in the abstract and introduction, that "B-BP suffers from poor robustness and sensitivity to noise and adversarial attacks". But the paper cites the Lin et. al 2024 paper to support this claim even though the Lin et. al 2024 paper does not mention B-BP at all, it instead discusses unrelated associative memories. So the criticisms of B-BP are lack support and the author(s) appear to confuse B-BP wi
1. The paper's strongest contribution is its scientific method. It proposes a robust gradient-free algorithm (B-SRA), performs a root-cause analysis to determine why it's robust (OWM + GPA), and then successfully ports those principles to fix the vulnerable B-BP algorithm. 2. The ablation in Sec 4.3.2 is excellent. It cleanly isolates the individual contributions of OWM (the ORTH strategy) and GPA (the ALIGN strategy) and demonstrates that both are required for full robustness (the SAME strategy
1. The paper focuses exclusively on Bidirectional Associative Memory (BAM), which is a classic but relatively niche architecture. The authors state an intent to apply these principles to Transformers and modern Hopfield networks as future work, but the paper presents no evidence that these findings will transfer. 2. The experiments use low-resolution, bipolarized images (MNIST, Chinese script) . While standard for testing associative memory, this is far from the complex, high-dimensional data wh
- While previous works introduced the Subspace Rotation Algorithm (SRA) for Restricted Hopfield Networks (RHN), this applies SRA to BAMs. - The algorithm is well explained and easy to implement with pseudo code - The authors also propose gradient pattern alignment (GPA) for aligning the gradient of the loss with the stored input patterns. Previous works do not apply GPA to associative memory training. The authors jointly apply Orthogonal Weight Matrix (OWM) regularization and GPA. - Evaluation
- Positioning - Need more clarity on the contribution. The work is an adaptation of SRA to BAM Orthogonality and gradient-input alignment style terms exist in broader literature; using them for training of BAM is reasonable but also incremental. - No direct comparisons to Dense Associative Memories / Modern Hopfield Networks or to orthogonality-promoting training in neural networks. - The authors claim that B-SRA enhances the robustness and convergence speed. But, do not provide any timing or
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Stochastic Gradient Optimization Techniques · Advanced Graph Neural Networks