Enhanced Scene Specificity with Sparse Dynamic Value Estimation

TL;DR

This paper introduces a sparse dynamic value estimation method for multi-scene reinforcement learning, reducing variance and improving performance by enforcing sparse scene-specific value clusters, leading to better rewards and navigation efficiency.

Contribution

It proposes a novel sparse clustering approach for dynamic value estimation in multi-scene RL, enhancing value function accuracy and agent performance.

Findings

Significant improvements in final reward scores across ProcGen environments.

Increased navigation efficiency in game level completion.

Reduced policy gradient variance through sparse clustering.

Abstract

Multi-scene reinforcement learning involves training the RL agent across multiple scenes / levels from the same task, and has become essential for many generalization applications. However, the inclusion of multiple scenes leads to an increase in sample variance for policy gradient computations, often resulting in suboptimal performance with the direct application of traditional methods (e.g. PPO, A3C). One strategy for variance reduction is to consider each scene as a distinct Markov decision process (MDP) and learn a joint value function dependent on both state (s) and MDP (M). However, this is non-trivial as the agent is usually unaware of the underlying level at train / test times in multi-scene RL. Recently, Singh et al. [1] tried to address this by proposing a dynamic value estimation approach that models the true joint value function distribution as a Gaussian mixture model (GMM). In this paper, we argue that the error between the true scene-specific value function and the predicted dynamic estimate can be further reduced by progressively enforcing sparse cluster assignments once the agent has explored most of the state space. The resulting agents not only show significant improvements in the final reward score across a range of OpenAI ProcGen environments, but also exhibit increased navigation efficiency while completing a game level.