Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation

TL;DR

This paper introduces CPPO, an on-policy contrastive reinforcement learning algorithm that learns goal-conditioned policies without reward functions, excelling in both continuous and discrete environments.

Contribution

The paper presents CPPO, the first on-policy contrastive RL method that directly derives policy advantages from contrastive Q-values, bridging a gap in existing CRL research.

Findings

CPPO outperforms previous CRL methods in 14 out of 18 tasks.

CPPO matches or exceeds PPO's performance in 12 out of 18 tasks.

CPPO works effectively in both continuous and discrete, single-agent and multi-agent environments.

Abstract

Contrastive reinforcement learning (CRL) learns goal-conditioned Q-values through a contrastive objective over state-action and goal representations, removing the need for hand-crafted reward functions. Despite impressive success in achieving viable self-supervised learning in RL, all existing CRL algorithms rely on off-policy optimisation and are mostly constrained to continuous action spaces, with little research invested in discrete environments. This leaves CRL disconnected from widely used and effective, modern on-policy training pipelines adopted across both single-agent and multi-agent RL in continuous and discrete environments. To establish a first connection, we introduce Contrastive Proximal Policy Optimisation (CPPO). CPPO is an on-policy contrastive RL algorithm that derives policy advantages directly from contrastive Q-values and optimises them via the standard PPO objective, without requiring a reward function or a replay buffer. We evaluate CPPO across continuous and discrete, single-agent and cooperative multi-agent tasks. Whilst the existence of an on-policy approach is inherently useful, we observe that \textbf{CPPO not only significantly outperforms the previous CRL baselines in 14 out of 18 tasks, but also matches or exceeds PPO's performance, which uses hand-crafted dense rewards, in 12 out of the 18 tasks tested.}