An Analytical Update Rule for General Policy Optimization

TL;DR

This paper introduces an analytical policy update rule for general stochastic policies that guarantees monotonic improvement, is independent of parametric approximators, and can be applied to off-policy and multi-agent reinforcement learning.

Contribution

It derives a new trust-region policy update rule from a closed-form solution, connecting policy search and value methods, and extends to multi-agent systems.

Findings

Provides a monotonic policy improvement guarantee.

Enables off-policy reinforcement learning without on-policy state integration.

Extends naturally to cooperative multi-agent systems.

Abstract

We present an analytical policy update rule that is independent of parametric function approximators. The policy update rule is suitable for optimizing general stochastic policies and has a monotonic improvement guarantee. It is derived from a closed-form solution to trust-region optimization using calculus of variation, following a new theoretical result that tightens existing bounds for policy improvement using trust-region methods. The update rule builds a connection between policy search methods and value function methods. Moreover, off-policy reinforcement learning algorithms can be derived from the update rule since it does not need to compute integration over on-policy states. In addition, the update rule extends immediately to cooperative multi-agent systems when policy updates are performed by one agent at a time.