MRS: Multi-Resolution Skills for HRL Agents

TL;DR

This paper introduces Multi-Resolution Skills (MRS), a hierarchical reinforcement learning approach that learns multiple goal-prediction modules at different temporal horizons to improve local subgoal selection and overall performance.

Contribution

MRS is the first method to learn multiple goal-prediction modules with a meta-controller for adaptive subgoal resolution in HRL, addressing reachability and temporal distance issues.

Findings

MRS outperforms fixed-resolution baselines on multiple benchmarks.

MRS reduces the performance gap between HRL and non-HRL methods.

MRS improves long-horizon task performance in complex environments.

Abstract

Hierarchical reinforcement learning (HRL) decomposes the policy into a manager and a worker, enabling long-horizon planning but introducing a performance gap on tasks requiring agility. We identify a root cause: in subgoal-based HRL, the manager's goal representation is typically learned without constraints on reachability or temporal distance from the current state, preventing precise local subgoal selection. We further show that the optimal subgoal distance is both task- and state-dependent: nearby subgoals enable precise control but amplify prediction noise, while distant subgoals produce smoother motion at the cost of geometric precision. We propose Multi-Resolution Skills (MRS), which learns multiple goal-prediction modules each specialized to a fixed temporal horizon, with a jointly trained meta-controller that selects among them based on the current state. MRS consistently outperforms fixed-resolution baselines and significantly reduces the performance gap between HRL and non-HRL state-of-the-art on DeepMind Control Suite, Gym-Robotics, and long-horizon AntMaze tasks. [Project page: https://sites.google.com/view/multi-res-skills/home]