EchoLSTM: A Self-Reflective Recurrent Network for Stabilizing Long-Range Memory

TL;DR

EchoLSTM introduces a self-reflective gating mechanism in recurrent networks, significantly improving long-range dependency modeling and robustness against noisy data, with competitive performance and higher efficiency.

Contribution

The paper presents EchoLSTM, a novel recurrent architecture with Output-Conditioned Gating that enhances memory stability through self-reflection, outperforming standard LSTMs and matching Transformer performance efficiently.

Findings

Achieves 69.0% accuracy on Distractor Signal Task, 33 points above baseline.

Matches Transformer performance on ListOps with fewer parameters.

Demonstrates increased robustness via qualitative trigger sensitivity analysis.

Abstract

Standard Recurrent Neural Networks, including LSTMs, struggle to model long-range dependencies, particularly in sequences containing noisy or misleading information. We propose a new architectural principle, Output-Conditioned Gating, which enables a model to perform self-reflection by modulating its internal memory gates based on its own past inferences. This creates a stabilizing feedback loop that enhances memory retention. Our final model, the EchoLSTM, integrates this principle with an attention mechanism. We evaluate the EchoLSTM on a series of challenging benchmarks. On a custom-designed Distractor Signal Task, the EchoLSTM achieves 69.0% accuracy, decisively outperforming a standard LSTM baseline by 33 percentage points. Furthermore, on the standard ListOps benchmark, the EchoLSTM achieves performance competitive with a modern Transformer model, 69.8% vs. 71.8%, while being over 5 times more parameter-efficient. A final Trigger Sensitivity Test provides qualitative evidence that our model's self-reflective mechanism leads to a fundamentally more robust memory system.