Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Trees

TL;DR

This paper introduces a novel framework enabling humans to specify initial robot behaviors using natural language, which are then converted into decision trees to warm-start reinforcement learning, improving accessibility and efficiency.

Contribution

The paper presents a new method to translate natural language into decision trees for initializing reinforcement learning policies, making autonomous systems more accessible to non-experts.

Findings

Achieved over 80% translation accuracy from natural language to decision trees.

Policies initialized with human language specifications match baseline RL performance.

Framework reduces domain exploration costs by leveraging natural language inputs.

Abstract

Human-AI policy specification is a novel procedure we define in which humans can collaboratively warm-start a robot's reinforcement learning policy. This procedure is comprised of two steps; (1) Policy Specification, i.e. humans specifying the behavior they would like their companion robot to accomplish, and (2) Policy Optimization, i.e. the robot applying reinforcement learning to improve the initial policy. Existing approaches to enabling collaborative policy specification are often unintelligible black-box methods, and are not catered towards making the autonomous system accessible to a novice end-user. In this paper, we develop a novel collaborative framework to allow humans to initialize and interpret an autonomous agent's behavior. Through our framework, we enable humans to specify an initial behavior model via unstructured, natural language (NL), which we convert to lexical decision trees. Next, we leverage these translated specifications, to warm-start reinforcement learning and allow the agent to further optimize these potentially suboptimal policies. Our approach warm-starts an RL agent by utilizing non-expert natural language specifications without incurring the additional domain exploration costs. We validate our approach by showing that our model is able to produce >80% translation accuracy, and that policies initialized by a human can match the performance of relevant RL baselines in two domains.