Discovering Hierarchical Achievements in Reinforcement Learning via Contrastive Learning

TL;DR

This paper introduces a contrastive learning approach called achievement distillation that enhances reinforcement learning agents' ability to discover hierarchical achievements efficiently, outperforming prior methods in complex environments.

Contribution

It presents a novel contrastive learning method that improves hierarchical achievement discovery in reinforcement learning using a simple, model-free algorithm with better efficiency and fewer parameters.

Findings

PPO with recent practices outperforms previous methods.

The agent can predict next achievements with limited confidence.

Achievement distillation achieves state-of-the-art results on Crafter environment.

Abstract

Discovering achievements with a hierarchical structure in procedurally generated environments presents a significant challenge. This requires an agent to possess a broad range of abilities, including generalization and long-term reasoning. Many prior methods have been built upon model-based or hierarchical approaches, with the belief that an explicit module for long-term planning would be advantageous for learning hierarchical dependencies. However, these methods demand an excessive number of environment interactions or large model sizes, limiting their practicality. In this work, we demonstrate that proximal policy optimization (PPO), a simple yet versatile model-free algorithm, outperforms previous methods when optimized with recent implementation practices. Moreover, we find that the PPO agent can predict the next achievement to be unlocked to some extent, albeit with limited confidence. Based on this observation, we introduce a novel contrastive learning method, called achievement distillation, which strengthens the agent's ability to predict the next achievement. Our method exhibits a strong capacity for discovering hierarchical achievements and shows state-of-the-art performance on the challenging Crafter environment in a sample-efficient manner while utilizing fewer model parameters.