Mix and Mask Actor-Critic Methods

TL;DR

This paper introduces a novel feature-sharing framework for actor-critic methods that dynamically couples and decouples shared representations, improving stability and efficiency in reinforcement learning.

Contribution

The paper proposes mix and mask mechanisms along with distributional scalarization to enhance shared feature learning in actor-critic algorithms.

Findings

Significant performance improvements over separate and shared backbone networks.

Effective dynamic coupling and decoupling of features between policy and value functions.

Enhanced stability and sample efficiency in reinforcement learning tasks.

Abstract

Shared feature spaces for actor-critic methods aims to capture generalized latent representations to be used by the policy and value function with the hopes for a more stable and sample-efficient optimization. However, such a paradigm present a number of challenges in practice, as parameters generating a shared representation must learn off two distinct objectives, resulting in competing updates and learning perturbations. In this paper, we present a novel feature-sharing framework to address these difficulties by introducing the mix and mask mechanisms and the distributional scalarization technique. These mechanisms behaves dynamically to couple and decouple connected latent features variably between the policy and value function, while the distributional scalarization standardizes the two objectives using a probabilistic standpoint. From our experimental results, we demonstrate significant performance improvements compared to alternative methods using separate networks and networks with a shared backbone.