LongReward: Improving Long-context Large Language Models with AI Feedback

TL;DR

This paper introduces LongReward, a method that uses an off-the-shelf LLM to provide multi-dimensional rewards for long-context responses, improving the performance of large language models in long-context tasks through offline RL.

Contribution

The paper proposes LongReward, a novel reward mechanism for long-context LLMs using an off-the-shelf LLM and a new assessment pipeline, combined with offline RL to enhance long-context capabilities.

Findings

LongReward significantly improves long-context model performance.

It enhances models' ability to follow short instructions.

Long-context and short-context DPO can be combined without performance loss.

Abstract

Though significant advancements have been achieved in developing long-context large language models (LLMs), the compromised quality of LLM-synthesized data for supervised fine-tuning (SFT) often affects the long-context performance of SFT models and leads to inherent limitations. In principle, reinforcement learning (RL) with appropriate reward signals can further enhance models' capacities. However, how to obtain reliable rewards in long-context scenarios remains unexplored. To this end, we propose LongReward, a novel method that utilizes an off-the-shelf LLM to provide rewards for long-context model responses from four human-valued dimensions: helpfulness, logicality, faithfulness, and completeness, each with a carefully designed assessment pipeline. By combining LongReward and offline RL algorithm DPO, we are able to effectively improve long-context SFT models. Our experiments indicate that LongReward not only significantly improves models' long-context performance but also enhances their ability to follow short instructions. We also find that long-context DPO with LongReward and conventional short-context DPO can be used together without hurting either one's performance.