Transfer Learning Across Simulated Robots With Different Sensors

TL;DR

This paper explores transfer learning for simulated robots, enabling policies trained with expensive sensors to be adapted to use cheaper sensors like cameras, facilitating real-world deployment.

Contribution

It introduces a method combining Bootstrapped Dual Policy Iteration and Policy Shaping to transfer policies across different sensor modalities in simulation.

Findings

Successful transfer from proximity sensors to camera-based input

Policy maintains performance despite sensor change

Demonstrates feasibility of sensor-agnostic policy transfer

Abstract

For a robot to learn a good policy, it often requires expensive equipment (such as sophisticated sensors) and a prepared training environment conducive to learning. However, it is seldom possible to perfectly equip robots for economic reasons, nor to guarantee ideal learning conditions, when deployed in real-life environments. A solution would be to prepare the robot in the lab environment, when all necessary material is available to learn a good policy. After training in the lab, the robot should be able to get by without the expensive equipment that used to be available to it, and yet still be guaranteed to perform well on the field. The transition between the lab (source) and the real-world environment (target) is related to transfer learning, where the state-space between the source and target tasks differ. We tackle a simulated task with continuous states and discrete actions presenting this challenge, using Bootstrapped Dual Policy Iteration, a model-free actor-critic reinforcement learning algorithm, and Policy Shaping. Specifically, we train a BDPI agent, embodied by a virtual robot performing a task in the V-Rep simulator, sensing its environment through several proximity sensors. The resulting policy is then used by a second agent learning the same task in the same environment, but with camera images as input. The goal is to obtain a policy able to perform the task relying on merely camera images.