SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks

TL;DR

This paper introduces a novel offline reinforcement learning algorithm that combines ensemble Q-networks, gradient diversity penalties, and behavior cloning to improve stability, convergence speed, and performance on benchmark tasks.

Contribution

It presents a new model-free actor-critic method integrating ensemble Q-networks with penalties and behavior cloning, enhancing out-of-distribution action handling and training stability.

Findings

Achieves higher convergence speed and stability.

Outperforms existing methods on D4RL MuJoCo benchmarks.

Effectively suppresses overestimation of out-of-distribution actions.

Abstract

In this work, we build upon the offline reinforcement learning algorithm TD7, which incorporates State-Action Learned Embeddings (SALE) and a prioritized experience replay buffer (LAP). We propose a model-free actor-critic algorithm that integrates ensemble Q-networks and a gradient diversity penalty from EDAC. The ensemble Q-networks introduce penalties to guide the actor network toward in-distribution actions, effectively addressing the challenge of out-of-distribution actions. Meanwhile, the gradient diversity penalty encourages diverse Q-value gradients, further suppressing overestimation for out-of-distribution actions. Additionally, our method retains an adjustable behavior cloning (BC) term that directs the actor network toward dataset actions during early training stages, while gradually reducing its influence as the precision of the Q-ensemble improves. These enhancements work synergistically to improve the stability and precision of the training. Experimental results on the D4RL MuJoCo benchmarks demonstrate that our algorithm achieves higher convergence speed, stability, and performance compared to existing methods.