Optimizing TD3 for 7-DOF Robotic Arm Grasping: Overcoming Suboptimality with Exploration-Enhanced Contrastive Learning

TL;DR

This paper introduces EECL, a novel exploration method integrated into TD3 that enhances exploration for 7-DOF robotic arm control, leading to more optimal policies and faster convergence.

Contribution

The paper proposes EECL, a new exploration-enhanced contrastive learning module that improves exploration in TD3 for robotic manipulation tasks.

Findings

EECL significantly improves exploration efficiency.

EECL accelerates convergence speed of TD3.

EECL outperforms baseline TD3 in robotic grasping tasks.

Abstract

In actor-critic-based reinforcement learning algorithms such as Twin Delayed Deep Deterministic policy gradient (TD3), insufficient exploration of the spatial space can result in suboptimal policies when controlling 7-DOF robotic arms. To address this issue, we propose a novel Exploration-Enhanced Contrastive Learning (EECL) module that improves exploration by providing additional rewards for encountering novel states. Our module stores previously explored states in a buffer and identifies new states by comparing them with historical data using Euclidean distance within a K-dimensional tree (KDTree) framework. When the agent explores new states, exploration rewards are assigned. These rewards are then integrated into the TD3 algorithm, ensuring that the Q-learning process incorporates these signals, promoting more effective strategy optimization. We evaluate our method on the robosuite panda lift task, demonstrating that it significantly outperforms the baseline TD3 in terms of both efficiency and convergence speed in the tested environment.