Overcoming Overfitting in Reinforcement Learning via Gaussian Process Diffusion Policy

TL;DR

This paper introduces Gaussian Process Diffusion Policy (GPDP), a novel RL algorithm combining diffusion models and Gaussian Processes to improve adaptation and exploration under distribution shifts, reducing overfitting.

Contribution

The paper presents GPDP, integrating GPR with diffusion models for RL policies, enhancing exploration and robustness against distribution shifts, a novel approach in RL policy representation.

Findings

Outperforms state-of-the-art algorithms under distribution shift by 67.74% to 123.18%.

Maintains comparable performance under normal conditions.

Enhances exploration efficiency through kernel-based GPR.

Abstract

One of the key challenges that Reinforcement Learning (RL) faces is its limited capability to adapt to a change of data distribution caused by uncertainties. This challenge arises especially in RL systems using deep neural networks as decision makers or policies, which are prone to overfitting after prolonged training on fixed environments. To address this challenge, this paper proposes Gaussian Process Diffusion Policy (GPDP), a new algorithm that integrates diffusion models and Gaussian Process Regression (GPR) to represent the policy. GPR guides diffusion models to generate actions that maximize learned Q-function, resembling the policy improvement in RL. Furthermore, the kernel-based nature of GPR enhances the policy's exploration efficiency under distribution shifts at test time, increasing the chance of discovering new behaviors and mitigating overfitting. Simulation results on the Walker2d benchmark show that our approach outperforms state-of-the-art algorithms under distribution shift condition by achieving around 67.74% to 123.18% improvement in the RL's objective function while maintaining comparable performance under normal conditions.