On the Relationship Between Variational Inference and Auto-Associative Memory

TL;DR

This paper introduces a variational inference framework for auto-associative memories, unifying perceptual inference and memory retrieval, and compares different neural approaches on image datasets.

Contribution

It presents a novel variational inference formulation for auto-associative memories that integrates perceptual inference with memory retrieval in a unified mathematical framework.

Findings

Combines amortized and iterative inference methods for memory models

Evaluates proposed models on CIFAR10 and CLEVR datasets

Shows competitive performance with existing associative memory models

Abstract

In this article, we propose a variational inference formulation of auto-associative memories, allowing us to combine perceptual inference and memory retrieval into the same mathematical framework. In this formulation, the prior probability distribution onto latent representations is made memory dependent, thus pulling the inference process towards previously stored representations. We then study how different neural network approaches to variational inference can be applied in this framework. We compare methods relying on amortized inference such as Variational Auto Encoders and methods relying on iterative inference such as Predictive Coding and suggest combining both approaches to design new auto-associative memory models. We evaluate the obtained algorithms on the CIFAR10 and CLEVR image datasets and compare them with other associative memory models such as Hopfield Networks, End-to-End Memory Networks and Neural Turing Machines.