Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input

TL;DR

This paper introduces a method to improve reward learning from human preferences by incorporating feature-level feedback, inspired by pragmatic communication, leading to faster and more accurate reward model convergence.

Contribution

It proposes enriching preference queries with feature-level information and develops a learning approach that leverages this data for better reward inference.

Findings

Faster convergence to accurate rewards with fewer comparisons.

Effective in vision and language domains.

Validated with a real-world mushroom foraging experiment.

Abstract

Humans use social context to specify preferences over behaviors, i.e. their reward functions. Yet, algorithms for inferring reward models from preference data do not take this social learning view into account. Inspired by pragmatic human communication, we study how to extract fine-grained data regarding why an example is preferred that is useful for learning more accurate reward models. We propose to enrich binary preference queries to ask both (1) which features of a given example are preferable in addition to (2) comparisons between examples themselves. We derive an approach for learning from these feature-level preferences, both for cases where users specify which features are reward-relevant, and when users do not. We evaluate our approach on linear bandit settings in both vision- and language-based domains. Results support the efficiency of our approach in quickly converging to accurate rewards with fewer comparisons vs. example-only labels. Finally, we validate the real-world applicability with a behavioral experiment on a mushroom foraging task. Our findings suggest that incorporating pragmatic feature preferences is a promising approach for more efficient user-aligned reward learning.