Reinforcement Learning of Flexible Policies for Symbolic Instructions with Adjustable Mapping Specifications

TL;DR

This paper introduces SIAMS, a reinforcement learning framework that enables robots to adapt to flexible symbolic instructions with adjustable mappings, improving performance in inspection tasks with diverse conditions.

Contribution

The paper proposes a novel RL approach that separates symbolic instructions from their mappings, allowing for flexible and efficient learning of instructions with adjustable specifications.

Findings

Outperforms context-aware multitask RL in 3D simulations.

Effectively handles diversified instruction completion patterns.

Integrates linear temporal logic into RL for symbolic instruction representation.

Abstract

Symbolic task representation is a powerful tool for encoding human instructions and domain knowledge. Such instructions guide robots to accomplish diverse objectives and meet constraints through reinforcement learning (RL). Most existing methods are based on fixed mappings from environmental states to symbols. However, in inspection tasks, where equipment conditions must be evaluated from multiple perspectives to avoid errors of oversight, robots must fulfill the same symbol from different states. To help robots respond to flexible symbol mapping, we propose representing symbols and their mapping specifications separately within an RL policy. This approach imposes on RL policy to learn combinations of symbolic instructions and mapping specifications, requiring an efficient learning framework. To cope with this issue, we introduce an approach for learning flexible policies called Symbolic Instructions with Adjustable Mapping Specifications (SIAMS). This paper represents symbolic instructions using linear temporal logic (LTL), a formal language that can be easily integrated into RL. Our method addresses the diversified completion patterns of instructions by (1) a specification-aware state modulation, which embeds differences in mapping specifications in state features, and (2) a symbol-number-based task curriculum, which gradually provides tasks according to the learning's progress. Evaluations in 3D simulations with discrete and continuous action spaces demonstrate that our method outperforms context-aware multitask RL comparisons.