Optimizing the Long-Term Behaviour of Deep Reinforcement Learning for Pushing and Grasping

TL;DR

This paper evaluates the ability of deep reinforcement learning models, specifically VPG and Hourglass systems, to learn long-term rewards using a new bin sorting task, highlighting the importance of Double Q-Learning for accurate long-term predictions.

Contribution

The study demonstrates that combining Hourglass models with Double Q-Learning enables accurate long-term reward prediction in complex tasks, addressing limitations of previous methods.

Findings

Double Q-Learning is crucial for training with high discount factors.

Hourglass models can predict long-term action sequences effectively.

Discount factor scheduling and exploration methods have limited impact on performance.

Abstract

We investigate the "Visual Pushing for Grasping" (VPG) system by Zeng et al. and the "Hourglass" system by Ewerton et al., an evolution of the former. The focus of our work is the investigation of the capabilities of both systems to learn long-term rewards and policies. Zeng et al. original task only needs a limited amount of foresight. Ewerton et al. attain their best performance using an agent which only takes the most immediate action under consideration. We are interested in the ability of their models and training algorithms to accurately predict long-term Q-Values. To evaluate this ability, we design a new bin sorting task and reward function. Our task requires agents to accurately estimate future rewards and therefore use high discount factors in their Q-Value calculation. We investigate the behaviour of an adaptation of the VPG training algorithm on our task. We show that this adaptation can not accurately predict the required long-term action sequences. In addition to the limitations identified by Ewerton et al., it suffers from the known Deep Q-Learning problem of overestimated Q-Values. In an effort to solve our task, we turn to the Hourglass models and combine them with the Double Q-Learning approach. We show that this approach enables the models to accurately predict long-term action sequences when trained with large discount factors. Our results show that the Double Q-Learning technique is essential for training with very high discount factors, as the models Q-Value predictions diverge otherwise. We also experiment with different approaches for discount factor scheduling, loss calculation and exploration procedures. Our results show that the latter factors do not visibly influence the model's performance for our task.