Generalizing soft actor-critic algorithms to discrete action spaces

TL;DR

This paper introduces a discrete variant of the soft actor-critic algorithm, enabling off-policy learning in discrete action spaces, and demonstrates its superior performance on Atari games with reduced training time.

Contribution

It proposes a practical discrete soft actor-critic algorithm integrated into Rainbow, achieving state-of-the-art results with lower replay ratios and faster training.

Findings

SAC-BBF improves IQM from 1.045 to 1.088 over previous methods.

SAC-BBF trains in one-third the time of prior algorithms to reach similar performance.

SAC-BBF achieves super-human performance with low replay ratio 2.

Abstract

ATARI is a suite of video games used by reinforcement learning (RL) researchers to test the effectiveness of the learning algorithm. Receiving only the raw pixels and the game score, the agent learns to develop sophisticated strategies, even to the comparable level of a professional human games tester. Ideally, we also want an agent requiring very few interactions with the environment. Previous competitive model-free algorithms for the task use the valued-based Rainbow algorithm without any policy head. In this paper, we change it by proposing a practical discrete variant of the soft actor-critic (SAC) algorithm. The new variant enables off-policy learning using policy heads for discrete domains. By incorporating it into the advanced Rainbow variant, i.e., the ``bigger, better, faster'' (BBF), the resulting SAC-BBF improves the previous state-of-the-art interquartile mean (IQM) from 1.045 to 1.088, and it achieves these results using only replay ratio (RR) 2. By using lower RR 2, the training time of SAC-BBF is strictly one-third of the time required for BBF to achieve an IQM of 1.045 using RR 8. As a value of IQM greater than one indicates super-human performance, SAC-BBF is also the only model-free algorithm with a super-human level using only RR 2. The code is publicly available on GitHub at https://github.com/lezhang-thu/bigger-better-faster-SAC.