Reliable Policy Iteration: Performance Robustness Across Architecture and Environment Perturbations

TL;DR

This paper evaluates the robustness of Reliable Policy Iteration (RPI) in classical control tasks, demonstrating its consistent high performance and stability compared to other deep reinforcement learning algorithms under various perturbations.

Contribution

The paper provides empirical evidence that RPI maintains performance robustness across different neural network and environment perturbations, highlighting its potential as a reliable RL method.

Findings

RPI reaches near-optimal performance early in training.

RPI sustains high performance despite perturbations.

RPI outperforms DQN, Double DQN, DDPG, TD3, and PPO in robustness.

Abstract

In a recent work, we proposed Reliable Policy Iteration (RPI), that restores policy iteration's monotonicity-of-value-estimates property to the function approximation setting. Here, we assess the robustness of RPI's empirical performance on two classical control tasks -- CartPole and Inverted Pendulum -- under changes to neural network and environmental parameters. Relative to DQN, Double DQN, DDPG, TD3, and PPO, RPI reaches near-optimal performance early and sustains this policy as training proceeds. Because deep RL methods are often hampered by sample inefficiency, training instability, and hyperparameter sensitivity, our results highlight RPI's promise as a more reliable alternative.