Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory

TL;DR

Kanerva++ introduces a differentiable, block-allocated memory extension to the Kanerva Machine, inspired by human memory systems, improving performance in memory-conditional image generation tasks with a hierarchical Bayesian approach.

Contribution

It presents a novel hierarchical Bayesian memory allocation scheme with differentiable, locally contiguous memory blocks, simplifying memory writing and enhancing generative performance.

Findings

Achieved state-of-the-art likelihood on binarized MNIST and Omniglot.

Demonstrated improved performance on multiple image datasets.

Simplified memory writing process compared to original Kanerva Machine.

Abstract

Episodic and semantic memory are critical components of the human memory model. The theory of complementary learning systems (McClelland et al., 1995) suggests that the compressed representation produced by a serial event (episodic memory) is later restructured to build a more generalized form of reusable knowledge (semantic memory). In this work we develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory via a hierarchical latent variable model. We take inspiration from traditional heap allocation and extend the idea of locally contiguous memory to the Kanerva Machine, enabling a novel differentiable block allocated latent memory. In contrast to the Kanerva Machine, we simplify the process of memory writing by treating it as a fully feed forward deterministic process, relying on the stochasticity of the read key distribution to disperse information within the memory. We demonstrate that this allocation scheme improves performance in memory conditional image generation, resulting in new state-of-the-art conditional likelihood values on binarized MNIST (<=41.58 nats/image) , binarized Omniglot (<=66.24 nats/image), as well as presenting competitive performance on CIFAR10, DMLab Mazes, Celeb-A and ImageNet32x32.