Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning

TL;DR

This paper introduces a search-based method to improve credit assignment in offline reinforcement learning by combining human preferences and expert demonstrations, leading to better policy learning.

Contribution

It proposes Search-Based Preference Weighting (SPW), a novel scheme that unifies preferences and demonstrations for more accurate credit assignment in offline RL.

Findings

SPW improves credit assignment accuracy in offline RL.

Joint learning from preferences and demonstrations outperforms prior methods.

Effective on challenging robot manipulation tasks.

Abstract

Offline reinforcement learning refers to the process of learning policies from fixed datasets, without requiring additional environment interaction. However, it often relies on well-defined reward functions, which are difficult and expensive to design. Human feedback is an appealing alternative, but its two common forms, expert demonstrations and preferences, have complementary limitations. Demonstrations provide stepwise supervision, but they are costly to collect and often reflect limited expert behavior modes. In contrast, preferences are easier to collect, but it is unclear which parts of a behavior contribute most to a trajectory segment, leaving credit assignment unresolved. In this paper, we introduce a Search-Based Preference Weighting (SPW) scheme to unify these two feedback sources. For each transition in a preference labeled trajectory, SPW searches for the most similar state-action pairs from expert demonstrations and directly derives stepwise importance weights based on their similarity scores. These weights are then used to guide standard preference learning, enabling more accurate credit assignment that traditional approaches struggle to achieve. We demonstrate that SPW enables effective joint learning from preferences and demonstrations, outperforming prior methods that leverage both feedback types on challenging robot manipulation tasks.