Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning

TL;DR

This paper introduces CANOE, a framework combining synthetic data generation and reinforcement learning to enhance the contextual faithfulness of large language models across diverse tasks without human annotations.

Contribution

The paper presents a novel systematic framework, CANOE, that reduces hallucinations in LLMs by synthetic data and a rule-based reinforcement learning method called Dual-GRPO, eliminating the need for human-labeled reward data.

Findings

Significant improvement in faithfulness across 11 tasks.

Outperforms GPT-4o and OpenAI o1 in faithfulness metrics.

Effective reduction of hallucinations without human annotations.

Abstract

Teaching large language models (LLMs) to be faithful in the provided context is crucial for building reliable information-seeking systems. Therefore, we propose a systematic framework, CANOE, to reduce faithfulness hallucinations of LLMs across different downstream tasks without human annotations. Specifically, we first synthesize short-form question-answering (QA) data with four diverse tasks to construct high-quality and easily verifiable training data without human annotation. Also, we propose Dual-GRPO, a rule-based reinforcement learning method that includes three tailored rule-based rewards derived from synthesized short-form QA data, while simultaneously optimizing both short-form and long-form response generation. Notably, Dual-GRPO eliminates the need to manually label preference data to train reward models and avoids over-optimizing short-form generation when relying only on the synthesized short-form QA data. Experimental results show that CANOE greatly improves the faithfulness of LLMs across 11 different tasks, even outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.