Online Behavior Modification for Expressive User Control of RL-Trained Robots

TL;DR

This paper introduces a novel online behavior modification paradigm for RL-trained robots, enabling real-time user control over robot behavior, demonstrated through a new algorithm and user study showing improved user preference and robustness.

Contribution

The paper presents ACORD, a new algorithm for online behavior modification that enhances user control and expression in RL-trained robots, combining benefits of RL and shared autonomy.

Findings

ACORD allows real-time user control over robot behavior.

Users preferred ACORD's control and expression levels.

ACORD maintains RL robustness and autonomous execution.

Abstract

Reinforcement Learning (RL) is an effective method for robots to learn tasks. However, in typical RL, end-users have little to no control over how the robot does the task after the robot has been deployed. To address this, we introduce the idea of online behavior modification, a paradigm in which users have control over behavior features of a robot in real time as it autonomously completes a task using an RL-trained policy. To show the value of this user-centered formulation for human-robot interaction, we present a behavior diversity based algorithm, Adjustable Control Of RL Dynamics (ACORD), and demonstrate its applicability to online behavior modification in simulation and a user study. In the study (n=23) users adjust the style of paintings as a robot traces a shape autonomously. We compare ACORD to RL and Shared Autonomy (SA), and show ACORD affords user-preferred levels of control and expression, comparable to SA, but with the potential for autonomous execution and robustness of RL.