GMAC: A Distributional Perspective on Actor-Critic Framework

TL;DR

GMAC introduces a distributional actor-critic framework that effectively captures value distributions, addressing instability and sample conflation, and demonstrates improved performance in discrete and continuous environments.

Contribution

It proposes a novel distributional actor-critic method using Cramér distance and a Sample-Replacement algorithm, with Gaussian Mixture Model parameterization for enhanced efficiency.

Findings

GMAC accurately models value distributions.

It improves performance over traditional actor-critic methods.

The method is computationally efficient in various environments.

Abstract

In this paper, we devise a distributional framework on actor-critic as a solution to distributional instability, action type restriction, and conflation between samples and statistics. We propose a new method that minimizes the Cram\'er distance with the multi-step Bellman target distribution generated from a novel Sample-Replacement algorithm denoted SR($\lambda$), which learns the correct value distribution under multiple Bellman operations. Parameterizing a value distribution with Gaussian Mixture Model further improves the efficiency and the performance of the method, which we name GMAC. We empirically show that GMAC captures the correct representation of value distributions and improves the performance of a conventional actor-critic method with low computational cost, in both discrete and continuous action spaces using Arcade Learning Environment (ALE) and PyBullet environment.