Constraint Sampling Reinforcement Learning: Incorporating Expertise For Faster Learning

TL;DR

This paper introduces Constraint Sampling Reinforcement Learning (CSRL), a practical method that incorporates human expertise as constraints to accelerate learning in complex RL tasks, demonstrating faster policy development across multiple environments.

Contribution

The paper presents CSRL, an algorithm that adaptively combines multiple constraints to improve learning speed and robustness, applicable to various base RL algorithms.

Findings

CSRL outperforms baselines in four diverse environments.

Incorporating constraints accelerates early learning performance.

The method is effective with different base RL algorithms.

Abstract

Online reinforcement learning (RL) algorithms are often difficult to deploy in complex human-facing applications as they may learn slowly and have poor early performance. To address this, we introduce a practical algorithm for incorporating human insight to speed learning. Our algorithm, Constraint Sampling Reinforcement Learning (CSRL), incorporates prior domain knowledge as constraints/restrictions on the RL policy. It takes in multiple potential policy constraints to maintain robustness to misspecification of individual constraints while leveraging helpful ones to learn quickly. Given a base RL learning algorithm (ex. UCRL, DQN, Rainbow) we propose an upper confidence with elimination scheme that leverages the relationship between the constraints, and their observed performance, to adaptively switch among them. We instantiate our algorithm with DQN-type algorithms and UCRL as base algorithms, and evaluate our algorithm in four environments, including three simulators based on real data: recommendations, educational activity sequencing, and HIV treatment sequencing. In all cases, CSRL learns a good policy faster than baselines.