PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models

TL;DR

PrefCLM leverages crowdsourced large language models as simulated teachers in preference-based reinforcement learning, improving adaptability and user satisfaction in human-robot interaction scenarios.

Contribution

This paper introduces PrefCLM, a novel framework that uses crowdsourced LLMs and Dempster-Shafer Theory to enhance preference-based RL with human-in-the-loop refinement.

Findings

Achieves competitive performance with traditional methods

Facilitates more natural and efficient robot behaviors

Significantly improves user satisfaction in HRI

Abstract

Preference-based reinforcement learning (PbRL) is emerging as a promising approach to teaching robots through human comparative feedback, sidestepping the need for complex reward engineering. However, the substantial volume of feedback required in existing PbRL methods often lead to reliance on synthetic feedback generated by scripted teachers. This approach necessitates intricate reward engineering again and struggles to adapt to the nuanced preferences particular to human-robot interaction (HRI) scenarios, where users may have unique expectations toward the same task. To address these challenges, we introduce PrefCLM, a novel framework that utilizes crowdsourced large language models (LLMs) as simulated teachers in PbRL. We utilize Dempster-Shafer Theory to fuse individual preferences from multiple LLM agents at the score level, efficiently leveraging their diversity and collective intelligence. We also introduce a human-in-the-loop pipeline that facilitates collective refinements based on user interactive feedback. Experimental results across various general RL tasks show that PrefCLM achieves competitive performance compared to traditional scripted teachers and excels in facilitating more more natural and efficient behaviors. A real-world user study (N=10) further demonstrates its capability to tailor robot behaviors to individual user preferences, significantly enhancing user satisfaction in HRI scenarios.