The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG)

TL;DR

This paper introduces the FutureDial-RAG challenge at SLT 2024, focusing on retrieval augmented generation for dialog systems using a real-world customer service dataset, aiming to improve knowledge retrieval and response quality.

Contribution

It presents a new challenge with baseline systems and metrics for RAG in dialog systems, encouraging research on real-life knowledge retrieval and response generation.

Findings

Baseline systems reveal the difficulty of the tasks.

The challenge promotes research on RAG for practical dialog applications.

Metrics are designed to evaluate retrieval accuracy and response coherence.

Abstract

Recently, increasing research interests have focused on retrieval augmented generation (RAG) to mitigate hallucination for large language models (LLMs). Following this trend, we launch the FutureDial-RAG challenge at SLT 2024, which aims at promoting the study of RAG for dialog systems. The challenge builds upon the MobileCS2 dataset, a real-life customer service datasets with nearly 3000 high-quality dialogs containing annotations for knowledge base query and corresponding results. Over the dataset, we define two tasks, track 1 for knowledge retrieval and track 2 for response generation, which are core research questions in dialog systems with RAG. We build baseline systems for the two tracks and design metrics to measure whether the systems can perform accurate retrieval and generate informative and coherent response. The baseline results show that it is very challenging to perform well on the two tasks, which encourages the participating teams and the community to study how to make better use of RAG for real-life dialog systems.