Utilizing Skipped Frames in Action Repeats via Pseudo-Actions

TL;DR

This paper introduces pseudo-actions to utilize intermediate frames discarded during action repeats in reinforcement learning, improving sample efficiency by making more data available for training.

Contribution

The paper proposes a novel pseudo-action method that leverages intermediate frames in action repeats, enhancing data utilization in model-free reinforcement learning.

Findings

Improved sample efficiency in continuous control tasks.

Enhanced learning performance in discrete control tasks.

Compatible with various Q-learning algorithms.

Abstract

In many deep reinforcement learning settings, when an agent takes an action, it repeats the same action a predefined number of times without observing the states until the next action-decision point. This technique of action repetition has several merits in training the agent, but the data between action-decision points (i.e., intermediate frames) are, in effect, discarded. Since the amount of training data is inversely proportional to the interval of action repeats, they can have a negative impact on the sample efficiency of training. In this paper, we propose a simple but effective approach to alleviate to this problem by introducing the concept of pseudo-actions. The key idea of our method is making the transition between action-decision points usable as training data by considering pseudo-actions. Pseudo-actions for continuous control tasks are obtained as the average of the action sequence straddling an action-decision point. For discrete control tasks, pseudo-actions are computed from learned action embeddings. This method can be combined with any model-free reinforcement learning algorithm that involves the learning of Q-functions. We demonstrate the effectiveness of our approach on both continuous and discrete control tasks in OpenAI Gym.