Safety-Aware Task Composition for Discrete and Continuous Reinforcement Learning

TL;DR

This paper advances safe reinforcement learning by enabling Boolean composition of learned tasks with safety constraints in both discrete and continuous action spaces, supporting zero-shot policy composition.

Contribution

It introduces two safety notions, proves correctness under certain assumptions, and extends Boolean composition to continuous actions, enhancing safe RL capabilities.

Findings

Demonstrated safety-aware composition in grid world with value iteration.

Extended safety composition techniques to DQN with image observations.

Applied safety composition to continuous control with TD3 in Bullet physics environment.

Abstract

Compositionality is a critical aspect of scalable system design. Reinforcement learning (RL) has recently shown substantial success in task learning, but has only recently begun to truly leverage composition. In this paper, we focus on Boolean composition of learned tasks as opposed to functional or sequential composition. Existing Boolean composition for RL focuses on reaching a satisfying absorbing state in environments with discrete action spaces, but does not support composable safety (i.e., avoidance) constraints. We advance the state of the art in Boolean composition of learned tasks with three contributions: i) introduce two distinct notions of safety in this framework; ii) show how to enforce either safety semantics, prove correctness (under some assumptions), and analyze the trade-offs between the two safety notions; and iii) extend Boolean composition from discrete action spaces to continuous action spaces. We demonstrate these techniques using modified versions of value iteration in a grid world, Deep Q-Network (DQN) in a grid world with image observations, and Twin Delayed DDPG (TD3) in a continuous-observation and continuous-action Bullet physics environment. We believe that these contributions advance the theory of safe reinforcement learning by allowing zero-shot composition of policies satisfying safety properties.