A Composable Specification Language for Reinforcement Learning Tasks

TL;DR

This paper introduces a composable specification language for complex reinforcement learning tasks, enabling automatic reward shaping and improved learning efficiency through a new tool called SPECTRL.

Contribution

It presents a novel language for specifying RL tasks and an algorithm that compiles these specifications into reward functions with automatic shaping.

Findings

SPECTRL outperforms state-of-the-art baselines

Automatic reward shaping improves convergence

The language simplifies complex task specification

Abstract

Reinforcement learning is a promising approach for learning control policies for robot tasks. However, specifying complex tasks (e.g., with multiple objectives and safety constraints) can be challenging, since the user must design a reward function that encodes the entire task. Furthermore, the user often needs to manually shape the reward to ensure convergence of the learning algorithm. We propose a language for specifying complex control tasks, along with an algorithm that compiles specifications in our language into a reward function and automatically performs reward shaping. We implement our approach in a tool called SPECTRL, and show that it outperforms several state-of-the-art baselines.