Careful at Estimation and Bold at Exploration

TL;DR

This paper introduces a novel exploration strategy for deterministic policy reinforcement learning that mitigates aimless exploration and policy divergence by combining a double-Q function framework with a surrogate policy, leading to improved performance on Mujoco benchmarks.

Contribution

It proposes a new exploration method based on greedy Q softmax updates and surrogate policy learning, addressing key issues in policy-based exploration for DPRL.

Findings

Superior performance on Mujoco benchmarks.

Effective in complex environments like Humanoid.

Outperforms previous state-of-the-art methods.

Abstract

Exploration strategies in continuous action space are often heuristic due to the infinite actions, and these kinds of methods cannot derive a general conclusion. In prior work, it has been shown that policy-based exploration is beneficial for continuous action space in deterministic policy reinforcement learning(DPRL). However, policy-based exploration in DPRL has two prominent issues: aimless exploration and policy divergence, and the policy gradient for exploration is only sometimes helpful due to inaccurate estimation. Based on the double-Q function framework, we introduce a novel exploration strategy to mitigate these issues, separate from the policy gradient. We first propose the greedy Q softmax update schema for Q value update. The expected Q value is derived by weighted summing the conservative Q value over actions, and the weight is the corresponding greedy Q value. Greedy Q takes the maximum value of the two Q functions, and conservative Q takes the minimum value of the two different Q functions. For practicality, this theoretical basis is then extended to allow us to combine action exploration with the Q value update, except for the premise that we have a surrogate policy that behaves like this exploration policy. In practice, we construct such an exploration policy with a few sampled actions, and to meet the premise, we learn such a surrogate policy by minimizing the KL divergence between the target policy and the exploration policy constructed by the conservative Q. We evaluate our method on the Mujoco benchmark and demonstrate superior performance compared to previous state-of-the-art methods across various environments, particularly in the most complex Humanoid environment.