Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems

TL;DR

This paper demonstrates that knowledge augmented finetuning (KAFT) significantly improves factual accuracy in RAG and agent-based dialog systems by training LLMs with domain-specific data and knowledge, outperforming prompting methods.

Contribution

It introduces and empirically evaluates KAFT, a novel finetuning approach that enhances LLM performance in knowledge-intensive dialog tasks compared to traditional prompting.

Findings

KAFT outperforms prompting in factual accuracy in RAG and agent systems.

KAFT shows significant improvements on the MobileCS2 customer service dataset.

This is the first empirical study on the effectiveness of KAFT in dialog systems.

Abstract

Large language models (LLMs) have recently been applied to dialog systems. Despite making progress, LLMs are prone to errors in knowledge-intensive scenarios. Recently, approaches based on retrieval augmented generation (RAG) and agent have emerged to improve the factual accuracy by enhancing the LLMs with knowledge retrieved from external knowledge bases (KBs). This is mostly implemented by prompting the LLMs with instructions, examples and the retrieved knowledge. However, LLMs may have difficulty using the retrieved knowledge effectively for response generation, because they are not well trained to do such generation for specific domains. To mitigate this problem, we propose to finetune the LLMs in the RAG-based and agent-based systems with domain-specific data, together with domain-specific external knowledge, which is called knowledge augmented finetuning (KAFT). We base our study on the MobileCS2 dataset, a real-life customer service dialog dataset that features intensive knowledge interactions, to systematically compare the prompting and KAFT techniques in the RAG-based and agent-based systems. Experiment results show that KAFT substantially surpasses prompting in both RAG and agent systems, particularly in terms of factual accuracy. To the best of our knowledge, this paper represents the first solid empirical work to investigate the KAFT idea.