Intrinsically-Motivated Reinforcement Learning: A Brief Introduction

TL;DR

This paper introduces intrinsically-motivated reinforcement learning, focusing on improving exploration through intrinsic rewards, proposing a new entropy-based method, and demonstrating its effectiveness in simulations.

Contribution

It classifies existing intrinsic reward methods, analyzes their drawbacks, and proposes a novel Rényi entropy-based intrinsic reward that enhances exploration in RL.

Findings

The new intrinsic reward method outperforms existing strategies in simulations.

Proposed approach achieves higher exploration efficiency and robustness.

Extensive experiments validate the effectiveness of the Rényi entropy maximization.

Abstract

Reinforcement learning (RL) is one of the three basic paradigms of machine learning. It has demonstrated impressive performance in many complex tasks like Go and StarCraft, which is increasingly involved in smart manufacturing and autonomous driving. However, RL consistently suffers from the exploration-exploitation dilemma. In this paper, we investigated the problem of improving exploration in RL and introduced the intrinsically-motivated RL. In sharp contrast to the classic exploration strategies, intrinsically-motivated RL utilizes the intrinsic learning motivation to provide sustainable exploration incentives. We carefully classified the existing intrinsic reward methods and analyzed their practical drawbacks. Moreover, we proposed a new intrinsic reward method via R\'enyi state entropy maximization, which overcomes the drawbacks of the preceding methods and provides powerful exploration incentives. Finally, extensive simulation demonstrated that the proposed module achieve superior performance with higher efficiency and robustness.