Learning Dynamic Abstract Representations for Sample-Efficient Reinforcement Learning

TL;DR

This paper introduces a domain-independent method for dynamically learning state abstractions during reinforcement learning, significantly improving sample efficiency and performance across various problems.

Contribution

It proposes a novel top-down, dynamic abstraction method based on Q-value dispersion, reducing the need for manual abstraction design in RL.

Findings

Automatically learns problem-specific abstractions

Achieves higher sample efficiency

Outperforms existing approaches in multiple domains

Abstract

In many real-world problems, the learning agent needs to learn a problem's abstractions and solution simultaneously. However, most such abstractions need to be designed and refined by hand for different problems and domains of application. This paper presents a novel top-down approach for constructing state abstractions while carrying out reinforcement learning. Starting with state variables and a simulator, it presents a novel domain-independent approach for dynamically computing an abstraction based on the dispersion of Q-values in abstract states as the agent continues acting and learning. Extensive empirical evaluation on multiple domains and problems shows that this approach automatically learns abstractions that are finely-tuned to the problem, yield powerful sample efficiency, and result in the RL agent significantly outperforming existing approaches.