Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback

TL;DR

This paper introduces an interactive visualization tool and active learning approach for reinforcement learning from human feedback, enabling more efficient exploration and comparison of behaviors to improve AI alignment.

Contribution

It presents a novel interactive visualization interface combined with active learning to enhance data collection in RLHF, improving policy quality and reward outcomes.

Findings

Increased final rewards by 69.34% in six robotics tasks

Lower error rates in behavior comparison

Better policies achieved through the proposed method

Abstract

Reinforcement learning from human feedback (RLHF) has emerged as a key enabling technology for aligning AI behaviour with human preferences. The traditional way to collect data in RLHF is via pairwise comparisons: human raters are asked to indicate which one of two samples they prefer. We present an interactive visualisation that better exploits the human visual ability to compare and explore whole groups of samples. The interface is comprised of two linked views: 1) an exploration view showing a contextual overview of all sampled behaviours organised in a hierarchical clustering structure; and 2) a comparison view displaying two selected groups of behaviours for user queries. Users can efficiently explore large sets of behaviours by iterating between these two views. Additionally, we devised an active learning approach suggesting groups for comparison. As shown by our evaluation in six simulated robotics tasks, our approach increases the final rewards by 69.34%. It leads to lower error rates and better policies. We open-source the code that can be easily integrated into the RLHF training loop, supporting research on human-AI alignment.