Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction

TL;DR

This paper introduces Geometric Active Exploration (GAE), a novel algorithm that leverages geometric abstractions in Markov Decision Processes to improve the efficiency of experimental design in scientific discovery.

Contribution

It extends MDP homomorphisms to Convex RL and provides the first formal analysis of how abstraction benefits sample efficiency in active exploration.

Findings

GAE outperforms baseline methods in efficiency.

Abstraction via homomorphisms improves sample complexity.

Theoretical analysis confirms the benefits of geometric abstraction.

Abstract

How can a scientist use a Reinforcement Learning (RL) algorithm to design experiments over a dynamical system's state space? In the case of finite and Markovian systems, an area called Active Exploration (AE) relaxes the optimization problem of experiments design into Convex RL, a generalization of RL admitting a wider notion of reward. Unfortunately, this framework is currently not scalable and the potential of AE is hindered by the vastness of experiment spaces typical of scientific discovery applications. However, these spaces are often endowed with natural geometries, e.g., permutation invariance in molecular design, that an agent could leverage to improve the statistical and computational efficiency of AE. To achieve this, we bridge AE and MDP homomorphisms, which offer a way to exploit known geometric structures via abstraction. Towards this goal, we make two fundamental contributions: we extend MDP homomorphisms formalism to Convex RL, and we present, to the best of our knowledge, the first analysis that formally captures the benefit of abstraction via homomorphisms on sample efficiency. Ultimately, we propose the Geometric Active Exploration (GAE) algorithm, which we analyse theoretically and experimentally in environments motivated by problems in scientific discovery.