Risk Perspective Exploration in Distributional Reinforcement Learning

TL;DR

This paper introduces risk scheduling methods in distributional reinforcement learning to improve exploration by leveraging risk levels and optimistic behaviors, demonstrating enhanced performance in multi-agent scenarios.

Contribution

It proposes novel risk scheduling approaches for exploration in distributional RL and shows their effectiveness in multi-agent environments.

Findings

Risk scheduling improves exploration efficiency.

Enhanced performance of DMIX with risk-based exploration.

Effective in multi-agent control tasks.

Abstract

Distributional reinforcement learning demonstrates state-of-the-art performance in continuous and discrete control settings with the features of variance and risk, which can be used to explore. However, the exploration method employing the risk property is hard to find, although numerous exploration methods in Distributional RL employ the variance of return distribution per action. In this paper, we present risk scheduling approaches that explore risk levels and optimistic behaviors from a risk perspective. We demonstrate the performance enhancement of the DMIX algorithm using risk scheduling in a multi-agent setting with comprehensive experiments.