A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

TL;DR

This paper introduces a self-supervised reinforcement learning method for fine-tuning large language models by leveraging cross-attention signals within the model, eliminating the need for human feedback and improving prompt relevance.

Contribution

The novel approach uses cross-attention signals to derive reward signals for RL fine-tuning, reducing reliance on human-labeled data and enabling scalable model alignment.

Findings

Significant improvements in prompt relevance and coherence over non-RL baselines.

Outperforms standard policy gradient methods and synthetic preference-based RL.

Highlights potential for scaling alignment with minimal human supervision.

Abstract

We propose a novel reinforcement learning framework for post training large language models that does not rely on human in the loop feedback. Instead, our approach uses cross attention signals within the model itself to derive a self supervised reward, thereby guiding iterative fine tuning of the model policy. By analyzing how the model attends to the input prompt during generation, we construct measures of prompt coverage, focus, and coherence. We then use these measures to rank or score candidate responses, providing a reward signal that encourages the model to produce well aligned, on topic text. In empirical comparisons against standard policy gradient methods and RL fine tuning with synthetic preference models, our method shows significant gains in prompt relevance and consistency over a non RL baseline. While it does not yet match the performance of fully human supervised RLHF systems, it highlights an important direction for scaling alignment with minimal human labeling. We provide a detailed analysis, discuss potential limitations, and outline future work for combining cross-attention based signals with smaller amounts of human feedback.