GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback

TL;DR

This paper introduces GAIRL, a novel reinforcement learning method that combines demonstrations and human feedback to improve policy learning, outperforming traditional GAIL in various control tasks.

Contribution

The paper proposes GAIRL, integrating GAIL and interactive reinforcement learning, to surpass demonstration limits and enhance policy stability in complex tasks.

Findings

GAIRL outperforms GAIL in all tested tasks.

GAIRL learns more stable and near-optimal policies.

Combining demonstrations with human feedback is effective.

Abstract

Deep reinforcement learning (DRL) has achieved great successes in many simulated tasks. The sample inefficiency problem makes applying traditional DRL methods to real-world robots a great challenge. Generative Adversarial Imitation Learning (GAIL) -- a general model-free imitation learning method, allows robots to directly learn policies from expert trajectories in large environments. However, GAIL shares the limitation of other imitation learning methods that they can seldom surpass the performance of demonstrations. In this paper, to address the limit of GAIL, we propose GAN-Based Interactive Reinforcement Learning (GAIRL) from demonstration and human evaluative feedback by combining the advantages of GAIL and interactive reinforcement learning. We tested our proposed method in six physics-based control tasks, ranging from simple low-dimensional control tasks -- Cart Pole and Mountain Car, to difficult high-dimensional tasks -- Inverted Double Pendulum, Lunar Lander, Hopper and HalfCheetah. Our results suggest that with both optimal and suboptimal demonstrations, a GAIRL agent can always learn a more stable policy with optimal or close to optimal performance, while the performance of the GAIL agent is upper bounded by the performance of demonstrations or even worse than it. In addition, our results indicate the reason that GAIRL is superior over GAIL is the complementary effect of demonstrations and human evaluative feedback.