Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems

TL;DR

This paper introduces Umbrella Reinforcement Learning, a novel and efficient method combining umbrella sampling with neural network policy gradients to solve complex nonlinear RL problems more effectively than existing algorithms.

Contribution

It presents a new approach that integrates umbrella sampling into RL using neural networks, enhancing computational efficiency and exploration capabilities.

Findings

Outperforms state-of-the-art algorithms in hard RL tasks

Efficiently handles sparse rewards and state traps

Utilizes ensemble agents with modified rewards for better exploration

Abstract

We report a novel, computationally efficient approach for solving hard nonlinear problems of reinforcement learning (RL). Here we combine umbrella sampling, from computational physics/chemistry, with optimal control methods. The approach is realized on the basis of neural networks, with the use of policy gradient. It outperforms, by computational efficiency and implementation universality, all available state-of-the-art algorithms, in application to hard RL problems with sparse reward, state traps and lack of terminal states. The proposed approach uses an ensemble of simultaneously acting agents, with a modified reward which includes the ensemble entropy, yielding an optimal exploration-exploitation balance.