Self-Attentive Associative Memory

TL;DR

This paper introduces Self-attentive Associative Memory (SAM), a novel neural memory architecture that separates item and relational memories, enabling high-order relationship modeling and improving performance across diverse tasks.

Contribution

The paper proposes a new SAM operator that constructs high-order relational memories from item memories, enhancing neural networks' ability to perform relational reasoning.

Findings

Achieves competitive results on synthetic and real-world tasks

Effectively models high-order relationships between memory items

Improves reasoning capabilities in various machine learning domains

Abstract

Heretofore, neural networks with external memory are restricted to single memory with lossy representations of memory interactions. A rich representation of relationships between memory pieces urges a high-order and segregated relational memory. In this paper, we propose to separate the storage of individual experiences (item memory) and their occurring relationships (relational memory). The idea is implemented through a novel Self-attentive Associative Memory (SAM) operator. Found upon outer product, SAM forms a set of associative memories that represent the hypothetical high-order relationships between arbitrary pairs of memory elements, through which a relational memory is constructed from an item memory. The two memories are wired into a single sequential model capable of both memorization and relational reasoning. We achieve competitive results with our proposed two-memory model in a diversity of machine learning tasks, from challenging synthetic problems to practical testbeds such as geometry, graph, reinforcement learning, and question answering.