State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

TL;DR

This paper investigates how different environment state representations influence reinforcement learning agents' ability to learn robotic grasping tasks and transfer policies from simulation to real robots, highlighting the benefits of task-specific and pre-trained representations.

Contribution

It introduces a continuum of state representations for robotic grasping and analyzes their impact on learning efficiency and sim2real transfer, emphasizing the advantages of pre-trained and task-informed states.

Findings

Numerical states enable performance comparable to non-learning baselines.

Pre-trained image-based representations outperform end-to-end trained models.

Task-specific knowledge in representations accelerates training and improves transfer success.

Abstract

Choosing an appropriate representation of the environment for the underlying decision-making process of the reinforcement learning agent is not always straightforward. The state representation should be inclusive enough to allow the agent to informatively decide on its actions and disentangled enough to simplify policy training and the corresponding sim2real transfer. Given this outlook, this work examines the effect of various representations in incentivizing the agent to solve a specific robotic task: antipodal and planar object grasping. A continuum of state representations is defined, starting from hand-crafted numerical states to encoded image-based representations, with decreasing levels of induced task-specific knowledge. The effects of each representation on the ability of the agent to solve the task in simulation and the transferability of the learned policy to the real robot are examined and compared against a model-based approach with complete system knowledge. The results show that reinforcement learning agents using numerical states can perform on par with non-learning baselines. Furthermore, we find that agents using image-based representations from pre-trained environment embedding vectors perform better than end-to-end trained agents, and hypothesize that separation of representation learning from reinforcement learning can benefit sim2real transfer. Finally, we conclude that incentivizing the state representation with task-specific knowledge facilitates faster convergence for agent training and increases success rates in sim2real robot control.