Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model

TL;DR

This paper introduces RL-SaLLM-F, a novel method for online preference-based reinforcement learning that uses large language models to generate self-augmented feedback, eliminating the need for privileged reward information and improving feedback quality.

Contribution

The paper proposes a new LLM-based feedback mechanism for online PbRL that addresses query ambiguity and enhances feedback reliability without relying on privileged information.

Findings

Self-augmented LLM feedback outperforms scripted teacher feedback.

The double-check mechanism improves feedback reliability.

Method achieves competitive results on MetaWorld benchmarks.

Abstract

Preference-based reinforcement learning (PbRL) provides a powerful paradigm to avoid meticulous reward engineering by learning rewards based on human preferences. However, real-time human feedback is hard to obtain in online tasks. Most work suppose there is a "scripted teacher" that utilizes privileged predefined reward to provide preference feedback. In this paper, we propose a RL Self-augmented Large Language Model Feedback (RL-SaLLM-F) technique that does not rely on privileged information for online PbRL. RL-SaLLM-F leverages the reflective and discriminative capabilities of LLM to generate self-augmented trajectories and provide preference labels for reward learning. First, we identify an failure issue in LLM-based preference discrimination, specifically "query ambiguity", in online PbRL. Then LLM is employed to provide preference labels and generate self-augmented imagined trajectories that better achieve the task goal, thereby enhancing the quality and efficiency of feedback. Additionally, a double-check mechanism is introduced to mitigate randomness in the preference labels, improving the reliability of LLM feedback. The experiment across multiple tasks in the MetaWorld benchmark demonstrates the specific contributions of each proposed module in RL-SaLLM-F, and shows that self-augmented LLM feedback can effectively replace the impractical "scripted teacher" feedback. In summary, RL-SaLLM-F introduces a new direction of feedback acquisition in online PbRL that does not rely on any online privileged information, offering an efficient and lightweight solution with LLM-driven feedback.