Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning

TL;DR

This paper presents a method for transferring policies from simulation to real-world robotics by learning image-to-semantics mappings, reducing annotation costs and improving transfer efficiency and interpretability.

Contribution

The authors introduce techniques like pair augmentation and active learning to reduce annotation costs in image-to-semantics translation for policy transfer in reinforcement learning.

Findings

Reduced annotation cost without performance loss

Outperformed existing methods without annotation

Improved efficiency and interpretability in sim-to-real transfer

Abstract

We investigate policy transfer using image-to-semantics translation to mitigate learning difficulties in vision-based robotics control agents. This problem assumes two environments: a simulator environment with semantics, that is, low-dimensional and essential information, as the state space, and a real-world environment with images as the state space. By learning mapping from images to semantics, we can transfer a policy, pre-trained in the simulator, to the real world, thereby eliminating real-world on-policy agent interactions to learn, which are costly and risky. In addition, using image-to-semantics mapping is advantageous in terms of the computational efficiency to train the policy and the interpretability of the obtained policy over other types of sim-to-real transfer strategies. To tackle the main difficulty in learning image-to-semantics mapping, namely the human annotation cost for producing a training dataset, we propose two techniques: pair augmentation with the transition function in the simulator environment and active learning. We observed a reduction in the annotation cost without a decline in the performance of the transfer, and the proposed approach outperformed the existing approach without annotation.