Learning Uncertainty-Aware Temporally-Extended Actions

TL;DR

This paper introduces UTE, a new reinforcement learning algorithm that uses uncertainty measurement to improve temporally-extended actions, leading to better policy learning in complex environments.

Contribution

The paper presents UTE, a novel uncertainty-aware algorithm that enhances action repetition by strategically balancing exploration and exploitation.

Findings

UTE outperforms existing action repetition methods.

UTE mitigates performance degradation caused by sub-optimal action repetition.

Experimental results show improved learning efficiency in Gridworld and Atari environments.

Abstract

In reinforcement learning, temporal abstraction in the action space, exemplified by action repetition, is a technique to facilitate policy learning through extended actions. However, a primary limitation in previous studies of action repetition is its potential to degrade performance, particularly when sub-optimal actions are repeated. This issue often negates the advantages of action repetition. To address this, we propose a novel algorithm named Uncertainty-aware Temporal Extension (UTE). UTE employs ensemble methods to accurately measure uncertainty during action extension. This feature allows policies to strategically choose between emphasizing exploration or adopting an uncertainty-averse approach, tailored to their specific needs. We demonstrate the effectiveness of UTE through experiments in Gridworld and Atari 2600 environments. Our findings show that UTE outperforms existing action repetition algorithms, effectively mitigating their inherent limitations and significantly enhancing policy learning efficiency.