Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions

TL;DR

This paper develops reinforcement learning algorithms that automatically learn and refine abstractions of parameterized action spaces, improving efficiency in complex decision-making tasks.

Contribution

It introduces algorithms that autonomously learn and refine state and action abstractions online, addressing limitations of existing methods in parameterized action spaces.

Findings

Achieves higher sample efficiency than state-of-the-art baselines.

Enables RL in long-horizon, sparse-reward settings with parameterized actions.

Refines abstractions during learning to focus on critical regions.

Abstract

Real-world sequential decision-making often involves parameterized action spaces that require both, decisions regarding discrete actions and decisions about continuous action parameters governing how an action is executed. Existing approaches exhibit severe limitations in this setting -- planning methods demand hand-crafted action models, and standard reinforcement learning (RL) algorithms are designed for either discrete or continuous actions but not both, and the few RL methods that handle parameterized actions typically rely on domain-specific engineering and fail to exploit the latent structure of these spaces. This paper extends the scope of RL algorithms to long-horizon, sparse-reward settings with parameterized actions by enabling agents to autonomously learn both state and action abstractions online. We introduce algorithms that progressively refine these abstractions during learning, increasing fine-grained detail in the critical regions of the state-action space where greater resolution improves performance. Across several continuous-state, parameterized-action domains, our abstraction-driven approach enables TD($\lambda$) to achieve markedly higher sample efficiency than state-of-the-art baselines.