# Augment-Reinforce-Merge Policy Gradient for Binary Stochastic Policy

## TL;DR

This paper introduces the ARM policy gradient estimator, a low-variance, unbiased method for binary stochastic policies that improves stability and convergence speed in on-policy reinforcement learning.

## Contribution

It proposes the ARM policy gradient estimator for binary actions, providing theoretical variance reduction guarantees and enhanced training stability.

## Key findings

- Achieves variance reduction with theoretical guarantees.
- Leads to more stable policy training.
- Enables faster convergence of neural network policies.

## Abstract

Due to the high variance of policy gradients, on-policy optimization algorithms are plagued with low sample efficiency. In this work, we propose Augment-Reinforce-Merge (ARM) policy gradient estimator as an unbiased low-variance alternative to previous baseline estimators on tasks with binary action space, inspired by the recent ARM gradient estimator for discrete random variable models. We show that the ARM policy gradient estimator achieves variance reduction with theoretical guarantees, and leads to significantly more stable and faster convergence of policies parameterized by neural networks.