Computationally Efficient Reinforcement Learning: Targeted Exploration leveraging Simple Rules

TL;DR

This paper introduces a simple modification to actor-critic reinforcement learning algorithms that uses expert rules to avoid suboptimal regions, significantly speeding up convergence without extra computational cost.

Contribution

It proposes a novel method to incorporate expert knowledge through action saturation and gradient modification, enhancing sample efficiency in continuous RL tasks.

Findings

Achieves 6-7x faster convergence in temperature control tasks.

Retains high final policy performance.

No additional computational overhead introduced.

Abstract

Model-free Reinforcement Learning (RL) generally suffers from poor sample complexity, mostly due to the need to exhaustively explore the state-action space to find well-performing policies. On the other hand, we postulate that expert knowledge of the system often allows us to design simple rules we expect good policies to follow at all times. In this work, we hence propose a simple yet effective modification of continuous actor-critic frameworks to incorporate such rules and avoid regions of the state-action space that are known to be suboptimal, thereby significantly accelerating the convergence of RL agents. Concretely, we saturate the actions chosen by the agent if they do not comply with our intuition and, critically, modify the gradient update step of the policy to ensure the learning process is not affected by the saturation step. On a room temperature control case study, it allows agents to converge to well-performing policies up to 6-7x faster than classical agents without computational overhead and while retaining good final performance.