ERPPO: Entropy Regularization-based Proximal Policy Optimization

TL;DR

ERPPO introduces an entropy regularization technique to improve multi-agent reinforcement learning by enhancing exploration and stability in high-ambiguity environments, leading to better object detection accuracy.

Contribution

The paper proposes ERPPO, a novel method that dynamically adjusts policy regularization based on observation ambiguity to improve multi-agent RL performance.

Findings

ERPPO outperforms MAPPO in accuracy and gradient magnitude.

ERPPO reduces false detections in uncertain visual conditions.

Experimental results confirm improved search success in maritime scenarios.

Abstract

Multi-Agent Proximal Policy Optimization (MAPPO) is a variant of the Proximal Policy Optimization (PPO) algorithm, specifically tailored for multi-agent reinforcement learning (MARL). MAPPO optimizes cooperative multi-agent settings by employing a centralized critic with decentralized actors. However, in case of multi-dimensional environment, MAPPO can not extract optimal policy due to non-stationary agent observation. To overcome this problem, we introduce a novel approach, Entropy Regularization-based Proximal Policy Optimization (ERPPO). For the policy optimization, we first define the object detection ambiguity under multi-dimensional observation environment. Distributional Spatiotemporal Ambiguity (DSA) learner is trained to estimate object detection uncertainty in non-stationary constraints. Then, we enhance PPO with a novel Entropy Regularization term. This regularization dynamically adjusts the policy update by applying a stronger (L1) regularization in high-ambiguity observation to encourage significant exploratory actions and a weaker (L2) regularization in low-ambiguity observation to stabilize the proximal policy optimization. This approach is designed to enhance the probability of successful object localization in time-critical operations by reducing detection failures and optimizing search policy. Experiments on a testbed with AirSim-based maritime searching scenarios show that the proposed ERPPO improves accuracy performance. Our proposed method improves higher gradient than MAPPO. Qualitative results confirm that ERPPO effectiveness in terms of suppressing false detection in visually uncertain conditions.