Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments

TL;DR

Logarithmic Memory Networks (LMNs) introduce a hierarchical logarithmic tree structure to efficiently model long-range sequences, significantly reducing memory and computation costs for resource-constrained environments in NLP and time series tasks.

Contribution

LMNs present a novel hierarchical architecture that reduces attention complexity from O(n^2) to O(log(n)), enabling efficient long-range sequence modeling with implicit positional encoding.

Findings

Reduced memory footprint and computational complexity.

Effective long-range sequence modeling in resource-limited settings.

Open-source implementation available on GitHub.

Abstract

Long-range sequence modeling is a crucial aspect of natural language processing and time series analysis. However, traditional models like Recurrent Neural Networks (RNNs) and Transformers suffer from computational and memory inefficiencies, especially when dealing with long sequences. This paper introduces Logarithmic Memory Networks (LMNs), a novel architecture that leverages a hierarchical logarithmic tree structure to efficiently store and retrieve past information. LMNs dynamically summarize historical context, significantly reducing the memory footprint and computational complexity of attention mechanisms from O(n2) to O(log(n)). The model employs a single-vector, targeted attention mechanism to access stored information, and the memory block construction worker (summarizer) layer operates in two modes: a parallel execution mode during training for efficient processing of hierarchical tree structures and a sequential execution mode during inference, which acts as a memory management system. It also implicitly encodes positional information, eliminating the need for explicit positional encodings. These features make LMNs a robust and scalable solution for processing long-range sequences in resource-constrained environments, offering practical improvements in efficiency and scalability. The code is publicly available under the MIT License on GitHub: https://github.com/AhmedBoin/LogarithmicMemory.