STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization

TL;DR

This paper introduces STRAPPER, a novel semi-supervised reinforcement learning method that addresses the 'similarity trap' in preference-based RL by combining self-training and peer regularization, improving reward learning for complex behaviors.

Contribution

It proposes a new approach to preference-based RL that overcomes the similarity trap using self-training and peer regularization, enhancing reward learning with less human effort.

Findings

Effective in learning locomotion behaviors

Improves reward confidence in semi-supervised settings

Addresses the similarity trap issue

Abstract

Preference-based reinforcement learning (PbRL) promises to learn a complex reward function with binary human preference. However, such human-in-the-loop formulation requires considerable human effort to assign preference labels to segment pairs, hindering its large-scale applications. Recent approache has tried to reuse unlabeled segments, which implicitly elucidates the distribution of segments and thereby alleviates the human effort. And consistency regularization is further considered to improve the performance of semi-supervised learning. However, we notice that, unlike general classification tasks, in PbRL there exits a unique phenomenon that we defined as similarity trap in this paper. Intuitively, human can have diametrically opposite preferredness for similar segment pairs, but such similarity may trap consistency regularization fail in PbRL. Due to the existence of similarity trap, such consistency regularization improperly enhances the consistency possiblity of the model's predictions between segment pairs, and thus reduces the confidence in reward learning, since the augmented distribution does not match with the original one in PbRL. To overcome such issue, we present a self-training method along with our proposed peer regularization, which penalizes the reward model memorizing uninformative labels and acquires confident predictions. Empirically, we demonstrate that our approach is capable of learning well a variety of locomotion and robotic manipulation behaviors using different semi-supervised alternatives and peer regularization.