Product Kanerva Machines: Factorized Bayesian Memory

TL;DR

The paper introduces the Product Kanerva Machine, a hierarchical Bayesian memory system that enhances capacity and abstraction by combining multiple smaller models, enabling unsupervised clustering and object-based image factorization.

Contribution

It proposes a novel hierarchical extension of the Kanerva Machine, improving scalability and feature abstraction in Bayesian memory models.

Findings

Exhibits unsupervised clustering capabilities

Discovers sparse and combinatorial memory patterns

Factorizes simple images into object components

Abstract

An ideal cognitively-inspired memory system would compress and organize incoming items. The Kanerva Machine (Wu et al, 2018) is a Bayesian model that naturally implements online memory compression. However, the organization of the Kanerva Machine is limited by its use of a single Gaussian random matrix for storage. Here we introduce the Product Kanerva Machine, which dynamically combines many smaller Kanerva Machines. Its hierarchical structure provides a principled way to abstract invariant features and gives scaling and capacity advantages over single Kanerva Machines. We show that it can exhibit unsupervised clustering, find sparse and combinatorial allocation patterns, and discover spatial tunings that approximately factorize simple images by object.