Leveraging weights signals -- Predicting and improving generalizability in reinforcement learning

TL;DR

This paper introduces a method to predict the generalizability of RL agents using internal weights and modifies the PPO algorithm to enhance their ability to perform well in unseen environments.

Contribution

It presents a novel approach to predict RL agents' generalizability from internal weights and integrates this into PPO to improve performance on new environments.

Findings

Improved PPO agents show higher generalizability scores.

Prediction of generalizability from weights is effective.

Enhanced PPO outperforms standard PPO in unseen environments.

Abstract

Generalizability of Reinforcement Learning (RL) agents (ability to perform on environments different from the ones they have been trained on) is a key problem as agents have the tendency to overfit to their training environments. In order to address this problem and offer a solution to increase the generalizability of RL agents, we introduce a new methodology to predict the generalizability score of RL agents based on the internal weights of the agent's neural networks. Using this prediction capability, we propose some changes in the Proximal Policy Optimization (PPO) loss function to boost the generalization score of the agents trained with this upgraded version. Experimental results demonstrate that our improved PPO algorithm yields agents with stronger generalizability compared to the original version.