An Empirical Study of Retrieval Augmented Generation with Chain-of-Thought

TL;DR

This paper evaluates the RAFT method, combining chain-of-thought, fine-tuning, and retrieval augmentation, to improve generative dialogue models' reasoning and information extraction across diverse datasets and languages.

Contribution

It introduces and empirically assesses RAFT, a novel approach integrating CoT, SFT, and RAG, addressing gaps in long-form QA and Chinese datasets.

Findings

RAFT improves reasoning accuracy in dialogue models.

Chain-of-thought enhances model performance significantly.

Effective across multiple languages and reasoning tasks.

Abstract

Since the launch of ChatGPT at the end of 2022, generative dialogue models represented by ChatGPT have quickly become essential tools in daily life. As user expectations increase, enhancing the capability of generative dialogue models to solve complex problems has become a focal point of current research. This paper delves into the effectiveness of the RAFT (Retrieval Augmented Fine-Tuning) method in improving the performance of Generative dialogue models. RAFT combines chain-of-thought with model supervised fine-tuning (SFT) and retrieval augmented generation (RAG), which significantly enhanced the model's information extraction and logical reasoning abilities. We evaluated the RAFT method across multiple datasets and analysed its performance in various reasoning tasks, including long-form QA and short-form QA tasks, tasks in both Chinese and English, and supportive and comparison reasoning tasks. Notably, it addresses the gaps in previous research regarding long-form QA tasks and Chinese datasets. Moreover, we also evaluate the benefit of the chain-of-thought (CoT) in the RAFT method. This work offers valuable insights for studies focused on enhancing the performance of generative dialogue models.