Iterative Amortized Policy Optimization

TL;DR

This paper introduces iterative amortized policy optimization, a flexible approach that improves policy performance in deep reinforcement learning by iteratively refining policy estimates, surpassing direct amortization methods on benchmark tasks.

Contribution

It proposes a novel iterative amortized optimization method for policy networks, enhancing performance and exploration in continuous control tasks.

Findings

Outperforms direct amortization in benchmark tasks

Provides better policy estimates and exploration

Demonstrates the effectiveness of iterative refinement

Abstract

Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control, enabling the estimation and sampling of high-value actions. From the variational inference perspective on RL, policy networks, when used with entropy or KL regularization, are a form of \textit{amortized optimization}, optimizing network parameters rather than the policy distributions directly. However, \textit{direct} amortized mappings can yield suboptimal policy estimates and restricted distributions, limiting performance and exploration. Given this perspective, we consider the more flexible class of \textit{iterative} amortized optimizers. We demonstrate that the resulting technique, iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.