Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data

TL;DR

This paper introduces Inverse-Q*, a novel token-level reinforcement learning framework that improves large language model alignment without requiring preference data or complex reward models, enhancing efficiency and stability.

Contribution

Inverse-Q* is a new method that estimates optimal policies directly from model responses, reducing reliance on human annotations and external supervision.

Findings

Matches or exceeds PPO in convergence speed

Effectively aligns responses with human preferences

Reduces need for preference data and hyper-parameter tuning

Abstract

Reinforcement Learning from Human Feedback (RLHF) has proven effective in aligning large language models with human intentions, yet it often relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and present challenges in sample efficiency and stability. In this paper, we introduce Inverse-Q*, an innovative framework that transcends traditional RL methods by optimizing token-level reinforcement learning without the need for additional reward or value models. Inverse-Q* leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model's responses, facilitating more granular and flexible policy shaping. Our approach reduces reliance on human annotation and external supervision, making it especially suitable for low-resource settings. We present extensive experimental results demonstrating that Inverse-Q* not only matches but potentially exceeds the effectiveness of PPO in terms of convergence speed and the alignment of model responses with human preferences. Our findings suggest that Inverse-Q* offers a practical and robust alternative to conventional RLHF approaches, paving the way for more efficient and adaptable model training approaches.