Beyond Goldfish Memory: Long-Term Open-Domain Conversation

TL;DR

This paper introduces a new long-term open-domain conversation dataset and demonstrates that retrieval-augmented and summarization-based models significantly outperform traditional models in maintaining context over multiple sessions.

Contribution

The authors provide a novel dataset for long-term conversations and evaluate models showing the superiority of retrieval-augmented and summarization techniques.

Findings

Retrieval-augmented models outperform standard architectures.

Models with summarization and recall capabilities perform better.

Long-term context handling is crucial for realistic dialogue systems.

Abstract

Despite recent improvements in open-domain dialogue models, state of the art models are trained and evaluated on short conversations with little context. In contrast, the long-term conversation setting has hardly been studied. In this work we collect and release a human-human dataset consisting of multiple chat sessions whereby the speaking partners learn about each other's interests and discuss the things they have learnt from past sessions. We show how existing models trained on existing datasets perform poorly in this long-term conversation setting in both automatic and human evaluations, and we study long-context models that can perform much better. In particular, we find retrieval-augmented methods and methods with an ability to summarize and recall previous conversations outperform the standard encoder-decoder architectures currently considered state of the art.