Differentiable Neural Computers with Memory Demon

TL;DR

This paper introduces a memory demon for Differentiable Neural Computers that enhances memory content management by maximizing mutual information, leading to improved performance in neural network architectures with external memory.

Contribution

The paper proposes a novel memory demon mechanism that implicitly modifies memory contents to optimize information retention in DNC architectures.

Findings

Memory demon improves information retention in DNCs.

Maximizing mutual information enhances memory content quality.

The approach leads to better performance in tasks requiring external memory.

Abstract

A Differentiable Neural Computer (DNC) is a neural network with an external memory which allows for iterative content modification via read, write and delete operations. We show that information theoretic properties of the memory contents play an important role in the performance of such architectures. We introduce a novel concept of memory demon to DNC architectures which modifies the memory contents implicitly via additive input encoding. The goal of the memory demon is to maximize the expected sum of mutual information of the consecutive external memory contents.