MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation

TL;DR

MemoChat introduces a method for fine-tuning large language models to use self-generated memos, significantly improving consistency in long-range open-domain conversations through a structured memorization and retrieval process.

Contribution

The paper presents a novel pipeline for instructing LLMs to utilize memos for maintaining conversation consistency over long distances, using a new training approach based on public datasets.

Findings

MemoChat outperforms strong baselines in multiple testing scenarios.

The approach enhances long-term conversation consistency.

Experiments validate the effectiveness of memo-based dialogue management.

Abstract

We propose MemoChat, a pipeline for refining instructions that enables large language models (LLMs) to effectively employ self-composed memos for maintaining consistent long-range open-domain conversations. We demonstrate a long-range open-domain conversation through iterative "memorization-retrieval-response" cycles. This requires us to carefully design tailored tuning instructions for each distinct stage. The instructions are reconstructed from a collection of public datasets to teach the LLMs to memorize and retrieve past dialogues with structured memos, leading to enhanced consistency when participating in future conversations. We invite experts to manually annotate a test set designed to evaluate the consistency of long-range conversations questions. Experiments on three testing scenarios involving both open-source and API-accessible chatbots at scale verify the efficacy of MemoChat, which outperforms strong baselines. Our codes, data and models are available here: https://github.com/LuJunru/MemoChat.