Efficient Exploration in Resource-Restricted Reinforcement Learning

TL;DR

This paper introduces a resource-aware exploration bonus for reinforcement learning in environments with non-replenishable resources, significantly improving sample efficiency by balancing exploration and resource consumption.

Contribution

It formalizes resource-restricted RL and proposes RAEB, a novel exploration method that reduces resource waste and enhances exploration efficiency.

Findings

RAEB outperforms existing exploration strategies in resource-restricted environments.

RAEB improves sample efficiency by up to ten times.

The method effectively balances exploration and resource conservation.

Abstract

In many real-world applications of reinforcement learning (RL), performing actions requires consuming certain types of resources that are non-replenishable in each episode. Typical applications include robotic control with limited energy and video games with consumable items. In tasks with non-replenishable resources, we observe that popular RL methods such as soft actor critic suffer from poor sample efficiency. The major reason is that, they tend to exhaust resources fast and thus the subsequent exploration is severely restricted due to the absence of resources. To address this challenge, we first formalize the aforementioned problem as a resource-restricted reinforcement learning, and then propose a novel resource-aware exploration bonus (RAEB) to make reasonable usage of resources. An appealing feature of RAEB is that, it can significantly reduce unnecessary resource-consuming trials while effectively encouraging the agent to explore unvisited states. Experiments demonstrate that the proposed RAEB significantly outperforms state-of-the-art exploration strategies in resource-restricted reinforcement learning environments, improving the sample efficiency by up to an order of magnitude.