Multi-trainer Interactive Reinforcement Learning System

TL;DR

This paper introduces a multi-trainer interactive reinforcement learning system that aggregates feedback from multiple imperfect trainers to improve agent training in reward-sparse environments, demonstrating superior accuracy and policy quality.

Contribution

The paper proposes a novel MTIRL system that effectively combines multiple trainers' binary feedback, enhancing training reliability and policy optimality in reinforcement learning.

Findings

Aggregation method outperforms majority, weighted, and Bayesian voting.

MTIRL achieves higher feedback accuracy.

Policy trained with review model is closer to optimal.

Abstract

Interactive reinforcement learning can effectively facilitate the agent training via human feedback. However, such methods often require the human teacher to know what is the correct action that the agent should take. In other words, if the human teacher is not always reliable, then it will not be consistently able to guide the agent through its training. In this paper, we propose a more effective interactive reinforcement learning system by introducing multiple trainers, namely Multi-Trainer Interactive Reinforcement Learning (MTIRL), which could aggregate the binary feedback from multiple non-perfect trainers into a more reliable reward for an agent training in a reward-sparse environment. In particular, our trainer feedback aggregation experiments show that our aggregation method has the best accuracy when compared with the majority voting, the weighted voting, and the Bayesian method. Finally, we conduct a grid-world experiment to show that the policy trained by the MTIRL with the review model is closer to the optimal policy than that without a review model.