C-Learning: Horizon-Aware Cumulative Accessibility Estimation

TL;DR

This paper introduces cumulative accessibility functions for multi-goal reinforcement learning, enabling horizon-aware planning, improved success rates, and reduced sample complexity in complex control tasks.

Contribution

It proposes a novel recurrence-based approach to estimate goal reachability within a horizon, addressing limitations of existing methods in sample efficiency and path diversity.

Findings

Outperforms state-of-the-art algorithms in success rate

Reduces sample complexity significantly

Enables multiple path planning based on horizon

Abstract

Multi-goal reaching is an important problem in reinforcement learning needed to achieve algorithmic generalization. Despite recent advances in this field, current algorithms suffer from three major challenges: high sample complexity, learning only a single way of reaching the goals, and difficulties in solving complex motion planning tasks. In order to address these limitations, we introduce the concept of cumulative accessibility functions, which measure the reachability of a goal from a given state within a specified horizon. We show that these functions obey a recurrence relation, which enables learning from offline interactions. We also prove that optimal cumulative accessibility functions are monotonic in the planning horizon. Additionally, our method can trade off speed and reliability in goal-reaching by suggesting multiple paths to a single goal depending on the provided horizon. We evaluate our approach on a set of multi-goal discrete and continuous control tasks. We show that our method outperforms state-of-the-art goal-reaching algorithms in success rate, sample complexity, and path optimality. Our code is available at https://github.com/layer6ai-labs/CAE, and additional visualizations can be found at https://sites.google.com/view/learning-cae/.