DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

TL;DR

This paper introduces DQN-TAMER, a reinforcement learning method that effectively incorporates real-time human feedback, addressing exploration challenges and demonstrating improved performance in simulated and real-world maze tasks.

Contribution

It proposes DQN-TAMER, a novel RL algorithm that combines human feedback with traditional rewards, and thoroughly models various human feedback scenarios.

Findings

DQN-TAMER outperforms baseline algorithms in Maze and Taxi environments.

The method effectively integrates human feedback with distant rewards.

Real-world application demonstrates facial expression recognition as feedback.

Abstract

Exploration has been one of the greatest challenges in reinforcement learning (RL), which is a large obstacle in the application of RL to robotics. Even with state-of-the-art RL algorithms, building a well-learned agent often requires too many trials, mainly due to the difficulty of matching its actions with rewards in the distant future. A remedy for this is to train an agent with real-time feedback from a human observer who immediately gives rewards for some actions. This study tackles a series of challenges for introducing such a human-in-the-loop RL scheme. The first contribution of this work is our experiments with a precisely modeled human observer: binary, delay, stochasticity, unsustainability, and natural reaction. We also propose an RL method called DQN-TAMER, which efficiently uses both human feedback and distant rewards. We find that DQN-TAMER agents outperform their baselines in Maze and Taxi simulated environments. Furthermore, we demonstrate a real-world human-in-the-loop RL application where a camera automatically recognizes a user's facial expressions as feedback to the agent while the agent explores a maze.