TOMA: Topological Map Abstraction for Reinforcement Learning

TL;DR

TOMA introduces a novel topological map abstraction method for reinforcement learning that creates efficient, robust graph representations of environments, enhancing exploration and planning performance.

Contribution

The paper presents TOMA, a new graph abstraction technique that reduces memory and computation costs and improves exploration in RL, outperforming existing methods.

Findings

TOMA generates more memory-efficient graph representations.

TOMA accelerates exploration by guiding agents to unexplored states.

TOMA achieves state-of-the-art performance in experiments.

Abstract

Animals are able to discover the topological map (graph) of surrounding environment, which will be used for navigation. Inspired by this biological phenomenon, researchers have recently proposed to generate graph representation for Markov decision process (MDP) and use such graphs for planning in reinforcement learning (RL). However, existing graph generation methods suffer from many drawbacks. One drawback is that existing methods do not learn an abstraction for graphs, which results in high memory and computation cost. This drawback also makes generated graph non-robust, which degrades the planning performance. Another drawback is that existing methods cannot be used for facilitating exploration which is important in RL. In this paper, we propose a new method, called topological map abstraction (TOMA), for graph generation. TOMA can generate an abstract graph representation for MDP, which costs much less memory and computation cost than existing methods. Furthermore, TOMA can be used for facilitating exploration. In particular, we propose planning to explore, in which TOMA is used to accelerate exploration by guiding the agent towards unexplored states. A novel experience replay module called vertex memory is also proposed to improve exploration performance. Experimental results show that TOMA can outperform existing methods to achieve the state-of-the-art performance.