Discrete and Continuous Action Representation for Practical RL in Video Games

TL;DR

This paper introduces Hybrid SAC, an RL algorithm capable of handling discrete, continuous, and parameterized actions, demonstrating its effectiveness in video game scenarios and benchmarking tasks.

Contribution

The paper presents Hybrid SAC, extending Soft Actor-Critic to support mixed action spaces, and explores the use of normalizing flows to enhance policy expressiveness.

Findings

Hybrid SAC successfully solves a high-speed driving task.

Hybrid SAC is competitive on parameterized action benchmarks.

Normalizing flows increase policy expressiveness with minimal computational cost.

Abstract

While most current research in Reinforcement Learning (RL) focuses on improving the performance of the algorithms in controlled environments, the use of RL under constraints like those met in the video game industry is rarely studied. Operating under such constraints, we propose Hybrid SAC, an extension of the Soft Actor-Critic algorithm able to handle discrete, continuous and parameterized actions in a principled way. We show that Hybrid SAC can successfully solve a highspeed driving task in one of our games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks. We also explore the impact of using normalizing flows to enrich the expressiveness of the policy at minimal computational cost, and identify a potential undesired effect of SAC when used with normalizing flows, that may be addressed by optimizing a different objective.