Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents

TL;DR

This paper introduces PILoT, a goal-planner that enables model-free skill transfer across diverse agents and tasks by planning immediate landmarks, reducing re-training and improving sample efficiency.

Contribution

PILoT is a novel goal-conditioned planning method that distills a universal goal-planner for cross-agent transfer in reinforcement learning.

Findings

Effective in few-shot transfer across different action spaces and dynamics.

Successful zero-shot transfer from simple to complex tasks.

Works with various input types, including vector states and images.

Abstract

In reinforcement learning applications like robotics, agents usually need to deal with various input/output features when specified with different state/action spaces by their developers or physical restrictions. This indicates unnecessary re-training from scratch and considerable sample inefficiency, especially when agents follow similar solution steps to achieve tasks. In this paper, we aim to transfer similar high-level goal-transition knowledge to alleviate the challenge. Specifically, we propose PILoT, i.e., Planning Immediate Landmarks of Targets. PILoT utilizes the universal decoupled policy optimization to learn a goal-conditioned state planner; then, distills a goal-planner to plan immediate landmarks in a model-free style that can be shared among different agents. In our experiments, we show the power of PILoT on various transferring challenges, including few-shot transferring across action spaces and dynamics, from low-dimensional vector states to image inputs, from simple robot to complicated morphology; and we also illustrate a zero-shot transfer solution from a simple 2D navigation task to the harder Ant-Maze task.