Nuclear Norm Maximization Based Curiosity-Driven Learning

TL;DR

This paper introduces a nuclear norm maximization approach for curiosity-driven reinforcement learning, effectively quantifying exploration novelty with high noise tolerance, leading to state-of-the-art results on benchmark environments.

Contribution

The paper proposes a novel curiosity method based on nuclear norm maximization that improves noise robustness and exploration accuracy in reinforcement learning.

Findings

Achieves human-normalized score of 1.09 on 26 Atari games with intrinsic reward.

Outperforms previous curiosity methods in benchmark tests.

Demonstrates robustness to environment stochasticity.

Abstract

To handle the sparsity of the extrinsic rewards in reinforcement learning, researchers have proposed intrinsic reward which enables the agent to learn the skills that might come in handy for pursuing the rewards in the future, such as encouraging the agent to visit novel states. However, the intrinsic reward can be noisy due to the undesirable environment's stochasticity and directly applying the noisy value predictions to supervise the policy is detrimental to improve the learning performance and efficiency. Moreover, many previous studies employ $\ell^2$ norm or variance to measure the exploration novelty, which will amplify the noise due to the square operation. In this paper, we address aforementioned challenges by proposing a novel curiosity leveraging the nuclear norm maximization (NNM), which can quantify the novelty of exploring the environment more accurately while providing high-tolerance to the noise and outliers. We conduct extensive experiments across a variety of benchmark environments and the results suggest that NNM can provide state-of-the-art performance compared with previous curiosity methods. On 26 Atari games subset, when trained with only intrinsic reward, NNM achieves a human-normalized score of 1.09, which doubles that of competitive intrinsic rewards-based approaches. Our code will be released publicly to enhance the reproducibility.