An agent design with goal reaching guarantees for enhancement of learning

TL;DR

This paper introduces a flexible algorithm that enhances reinforcement learning agents by ensuring goal-reaching guarantees while improving learning efficiency, supported by formal proofs and empirical experiments.

Contribution

The paper proposes a novel algorithm that augments existing agents with goal-reaching guarantees and demonstrates its effectiveness through formal proof and empirical validation.

Findings

The algorithm boosts learning efficiency.

It guarantees goal reaching with existing agents.

Empirical results show improved performance.

Abstract

Reinforcement learning is commonly concerned with problems of maximizing accumulated rewards in Markov decision processes. Oftentimes, a certain goal state or a subset of the state space attain maximal reward. In such a case, the environment may be considered solved when the goal is reached. Whereas numerous techniques, learning or non-learning based, exist for solving environments, doing so optimally is the biggest challenge. Say, one may choose a reward rate which penalizes the action effort. Reinforcement learning is currently among the most actively developed frameworks for solving environments optimally by virtue of maximizing accumulated reward, in other words, returns. Yet, tuning agents is a notoriously hard task as reported in a series of works. Our aim here is to help the agent learn a near-optimal policy efficiently while ensuring a goal reaching property of some basis policy that merely solves the environment. We suggest an algorithm, which is fairly flexible, and can be used to augment practically any agent as long as it comprises of a critic. A formal proof of a goal reaching property is provided. Comparative experiments on several problems under popular baseline agents provided an empirical evidence that the learning can indeed be boosted while ensuring goal reaching property.