Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping

TL;DR

This paper introduces Quantile QT-Opt, a distributional reinforcement learning algorithm for vision-based robotic grasping, demonstrating improved success rates, sample efficiency, and risk management capabilities in complex real-world tasks.

Contribution

It proposes Quantile QT-Opt, a novel distributional RL method for continuous control, and evaluates its effectiveness in robotic grasping, including risk-aware decision making and batch RL comparisons.

Findings

Q2-Opt outperforms existing methods in grasp success rate.

Q2-Opt is more sample efficient in real robotic tasks.

Distributional approach enables risk management in robotic control.

Abstract

The distributional perspective on reinforcement learning (RL) has given rise to a series of successful Q-learning algorithms, resulting in state-of-the-art performance in arcade game environments. However, it has not yet been analyzed how these findings from a discrete setting translate to complex practical applications characterized by noisy, high dimensional and continuous state-action spaces. In this work, we propose Quantile QT-Opt (Q2-Opt), a distributional variant of the recently introduced distributed Q-learning algorithm for continuous domains, and examine its behaviour in a series of simulated and real vision-based robotic grasping tasks. The absence of an actor in Q2-Opt allows us to directly draw a parallel to the previous discrete experiments in the literature without the additional complexities induced by an actor-critic architecture. We demonstrate that Q2-Opt achieves a superior vision-based object grasping success rate, while also being more sample efficient. The distributional formulation also allows us to experiment with various risk distortion metrics that give us an indication of how robots can concretely manage risk in practice using a Deep RL control policy. As an additional contribution, we perform batch RL experiments in our virtual environment and compare them with the latest findings from discrete settings. Surprisingly, we find that the previous batch RL findings from the literature obtained on arcade game environments do not generalise to our setup.