Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning

TL;DR

This paper introduces a reinforcement learning approach that combines offline pre-training and online fine-tuning to create adaptive human-machine interfaces capable of handling noisy, high-dimensional command signals for tasks like robotic navigation.

Contribution

It presents a novel RL algorithm with a new method for inferring user intent, improving interface adaptability with limited user data and noisy inputs.

Findings

Enhanced goal navigation success rate over baseline interfaces.

Effective denoising of user commands and shared autonomy.

Improved performance in simulated tasks like pushing and lunar lander.

Abstract

Adaptive interfaces can help users perform sequential decision-making tasks like robotic teleoperation given noisy, high-dimensional command signals (e.g., from a brain-computer interface). Recent advances in human-in-the-loop machine learning enable such systems to improve by interacting with users, but tend to be limited by the amount of data that they can collect from individual users in practice. In this paper, we propose a reinforcement learning algorithm to address this by training an interface to map raw command signals to actions using a combination of offline pre-training and online fine-tuning. To address the challenges posed by noisy command signals and sparse rewards, we develop a novel method for representing and inferring the user's long-term intent for a given trajectory. We primarily evaluate our method's ability to assist users who can only communicate through noisy, high-dimensional input channels through a user study in which 12 participants performed a simulated navigation task by using their eye gaze to modulate a 128-dimensional command signal from their webcam. The results show that our method enables successful goal navigation more often than a baseline directional interface, by learning to denoise user commands signals and provide shared autonomy assistance. We further evaluate on a simulated Sawyer pushing task with eye gaze control, and the Lunar Lander game with simulated user commands, and find that our method improves over baseline interfaces in these domains as well. Extensive ablation experiments with simulated user commands empirically motivate each component of our method.