A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Models

TL;DR

This survey reviews the evolution of language model-based dialogue systems, highlighting historical developments, recent advancements with pre-trained and large language models, and discussing future challenges and directions.

Contribution

It provides a systematic, chronological review of dialogue system evolution in relation to language model breakthroughs, offering insights into future research directions.

Findings

Categorizes dialogue system development stages aligned with LM breakthroughs

Highlights the impact of pre-trained and large language models on dialogue systems

Discusses open challenges and future research directions

Abstract

Dialogue systems (DS), including the task-oriented dialogue system (TOD) and the open-domain dialogue system (ODD), have always been a fundamental task in natural language processing (NLP), allowing various applications in practice. Owing to sophisticated training and well-designed model architecture, language models (LM) are usually adopted as the necessary backbone to build the dialogue system. Consequently, every breakthrough in LM brings about a shift in learning paradigm and research attention within dialogue system, especially the appearance of pre-trained language models (PLMs) and large language models (LLMs). In this paper, we take a deep look at the history of the dialogue system, especially its special relationship with the advancements of language models. Thus, our survey offers a systematic perspective, categorizing different stages in a chronological order aligned with LM breakthroughs, providing a comprehensive review of state-of-the-art research outcomes. What's more, we turn our attention to emerging topics and engage in a discussion on open challenges, providing valuable insights into the future directions for LLM-based dialogue systems. In summary, this survey delves into the dynamic interplay between language models and dialogue systems, unraveling the evolutionary path of this essential relationship. Through this exploration, we pave the way for a deeper comprehension of the field, guiding future developments in LM-based dialogue systems.