Cyclophobic Reinforcement Learning

TL;DR

This paper introduces a cyclophobic intrinsic reward for reinforcement learning that discourages redundant cycles, leading to more efficient exploration in complex, sparse-reward environments like MiniGrid and MiniHack.

Contribution

The paper proposes a novel cyclophobic intrinsic reward that avoids cycles, enhancing exploration efficiency in sparse-reward environments, and combines it with hierarchical representations for improved performance.

Findings

Outperforms previous methods in MiniGrid and MiniHack environments

Achieves higher sample efficiency in complex exploration tasks

Demonstrates the effectiveness of cycle avoidance in reinforcement learning

Abstract

In environments with sparse rewards, finding a good inductive bias for exploration is crucial to the agent's success. However, there are two competing goals: novelty search and systematic exploration. While existing approaches such as curiosity-driven exploration find novelty, they sometimes do not systematically explore the whole state space, akin to depth-first-search vs breadth-first-search. In this paper, we propose a new intrinsic reward that is cyclophobic, i.e., it does not reward novelty, but punishes redundancy by avoiding cycles. Augmenting the cyclophobic intrinsic reward with a sequence of hierarchical representations based on the agent's cropped observations we are able to achieve excellent results in the MiniGrid and MiniHack environments. Both are particularly hard, as they require complex interactions with different objects in order to be solved. Detailed comparisons with previous approaches and thorough ablation studies show that our newly proposed cyclophobic reinforcement learning is more sample efficient than other state of the art methods in a variety of tasks.