The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG)
Yucheng Cai, Si Chen, Yuxuan Wu, Yi Huang, Junlan Feng, Zhijian Ou

TL;DR
This paper introduces the FutureDial-RAG challenge at SLT 2024, focusing on retrieval augmented generation for dialog systems using a real-world customer service dataset, aiming to improve knowledge retrieval and response quality.
Contribution
It presents a new challenge with baseline systems and metrics for RAG in dialog systems, encouraging research on real-life knowledge retrieval and response generation.
Findings
Baseline systems reveal the difficulty of the tasks.
The challenge promotes research on RAG for practical dialog applications.
Metrics are designed to evaluate retrieval accuracy and response coherence.
Abstract
Recently, increasing research interests have focused on retrieval augmented generation (RAG) to mitigate hallucination for large language models (LLMs). Following this trend, we launch the FutureDial-RAG challenge at SLT 2024, which aims at promoting the study of RAG for dialog systems. The challenge builds upon the MobileCS2 dataset, a real-life customer service datasets with nearly 3000 high-quality dialogs containing annotations for knowledge base query and corresponding results. Over the dataset, we define two tasks, track 1 for knowledge retrieval and track 2 for response generation, which are core research questions in dialog systems with RAG. We build baseline systems for the two tracks and design metrics to measure whether the systems can perform accurate retrieval and generate informative and coherent response. The baseline results show that it is very challenging to perform…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSpeech and dialogue systems · Topic Modeling · AI in Service Interactions
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · travel james · Attention Is All You Need · Byte Pair Encoding · Softmax · Layer Normalization · WordPiece · Dropout · Attention Dropout · BART
