Enhancing RLHF with Human Gaze Modeling

TL;DR

This paper introduces methods that incorporate human gaze modeling into RLHF to improve training efficiency and reduce computational costs while maintaining or enhancing model performance.

Contribution

It presents novel gaze-aware reward models and gaze-based reward distribution techniques that leverage human gaze data to enhance RLHF.

Findings

Gaze-informed RLHF converges faster than traditional methods.

Gaze-based rewards maintain or slightly improve model performance.

Using gaze data reduces computational costs during training.

Abstract

Reinforcement Learning from Human Feedback (RLHF) aligns language models with human preferences but is computationally expensive. We explore two approaches that leverage human gaze modeling to enhance RLHF: (1) gaze-aware reward models and (2) gaze-based distribution of sparse rewards at token level. Our experiments demonstate that gaze-informed RLHF achieves faster convergence while maintaining or slightly improving performance, thus, reducing computational costs during policy optimization. These results show that human gaze provides a valuable and underused signal for policy optimization, pointing to a promising direction for improving RLHF efficiency.