CHARM: Considering Human Attributes for Reinforcement Modeling

TL;DR

This study explores how human attributes influence feedback patterns in reinforcement learning, revealing correlations with characteristics like experience and education, and improving feedback prediction accuracy.

Contribution

It introduces an exploratory analysis linking human traits to feedback behaviors and enhances feedback prediction models using human characteristics.

Findings

Feedback patterns correlate with human traits such as robot experience and education.

Human characteristics improve the accuracy of feedback value prediction.

Feedback is influenced by both task statistics and individual human attributes.

Abstract

Reinforcement Learning from Human Feedback has recently achieved significant success in various fields, and its performance is highly related to feedback quality. While much prior work acknowledged that human teachers' characteristics would affect human feedback patterns, there is little work that has closely investigated the actual effects. In this work, we designed an exploratory study investigating how human feedback patterns are associated with human characteristics. We conducted a public space study with two long horizon tasks and 46 participants. We found that feedback patterns are not only correlated with task statistics, such as rewards, but also correlated with participants' characteristics, especially robot experience and educational background. Additionally, we demonstrated that human feedback value can be more accurately predicted with human characteristics compared to only using task statistics. All human feedback and characteristics we collected, and codes for our data collection and predicting more accurate human feedback are available at https://github.com/AABL-Lab/CHARM