Aging Memories Generate More Fluent Dialogue Responses with Memory Augmented Neural Networks

TL;DR

This paper introduces memory dropout, a regularization technique for Memory Networks that ages redundant memories to improve dialogue response generation, achieving state-of-the-art results on benchmark datasets.

Contribution

It proposes a novel memory dropout method that ages redundant memories, reducing correlation and overfitting, leading to improved neural dialogue models.

Findings

Memory dropout improves response quality.

Achieves state-of-the-art results on benchmark datasets.

Reduces memory redundancy and overfitting.

Abstract

Memory Networks have emerged as effective models to incorporate Knowledge Bases (KB) into neural networks. By storing KB embeddings into a memory component, these models can learn meaningful representations that are grounded to external knowledge. However, as the memory unit becomes full, the oldest memories are replaced by newer representations. In this paper, we question this approach and provide experimental evidence that conventional Memory Networks store highly correlated vectors during training. While increasing the memory size mitigates this problem, this also leads to overfitting as the memory stores a large number of training latent representations. To address these issues, we propose a novel regularization mechanism named memory dropout which 1) Samples a single latent vector from the distribution of redundant memories. 2) Ages redundant memories thus increasing their probability of overwriting them during training. This fully differentiable technique allows us to achieve state-of-the-art response generation in the Stanford Multi-Turn Dialogue and Cambridge Restaurant datasets.