Benchmarking Deep Reinforcement Learning Algorithms for Vision-based Robotics

TL;DR

This paper benchmarks several state-of-the-art deep reinforcement learning algorithms on challenging vision-based robotics tasks, introducing strategies for HER implementation and attention-based feature extraction, providing valuable insights into their comparative performance.

Contribution

It presents the first comprehensive benchmarking of RL algorithms on two complex vision-based robotics environments, including novel strategies for HER and attention mechanisms.

Findings

HER strategies improve learning in single-goal environments.

Attention mechanisms enhance feature extraction from RGB images.

Benchmark results highlight the strengths and limitations of each algorithm.

Abstract

This paper presents a benchmarking study of some of the state-of-the-art reinforcement learning algorithms used for solving two simulated vision-based robotics problems. The algorithms considered in this study include soft actor-critic (SAC), proximal policy optimization (PPO), interpolated policy gradients (IPG), and their variants with Hindsight Experience replay (HER). The performances of these algorithms are compared against PyBullet's two simulation environments known as KukaDiverseObjectEnv and RacecarZEDGymEnv respectively. The state observations in these environments are available in the form of RGB images and the action space is continuous, making them difficult to solve. A number of strategies are suggested to provide intermediate hindsight goals required for implementing HER algorithm on these problems which are essentially single-goal environments. In addition, a number of feature extraction architectures are proposed to incorporate spatial and temporal attention in the learning process. Through rigorous simulation experiments, the improvement achieved with these components are established. To the best of our knowledge, such a benchmarking study is not available for the above two vision-based robotics problems making it a novel contribution in the field.