Contrastive Abstraction for Reinforcement Learning

TL;DR

This paper introduces contrastive abstraction learning, a method that uses contrastive learning and Hopfield networks to create abstract state representations, improving reinforcement learning efficiency without relying on reward signals.

Contribution

It proposes a novel two-phase approach combining contrastive learning and Hopfield networks to automatically discover abstract states in reinforcement learning tasks.

Findings

Effective abstraction improves RL performance

Method does not require reward signals

Demonstrated success on various tasks

Abstract

Learning agents with reinforcement learning is difficult when dealing with long trajectories that involve a large number of states. To address these learning problems effectively, the number of states can be reduced by abstract representations that cluster states. In principle, deep reinforcement learning can find abstract states, but end-to-end learning is unstable. We propose contrastive abstraction learning to find abstract states, where we assume that successive states in a trajectory belong to the same abstract state. Such abstract states may be basic locations, achieved subgoals, inventory, or health conditions. Contrastive abstraction learning first constructs clusters of state representations by contrastive learning and then applies modern Hopfield networks to determine the abstract states. The first phase of contrastive abstraction learning is self-supervised learning, where contrastive learning forces states with sequential proximity to have similar representations. The second phase uses modern Hopfield networks to map similar state representations to the same fixed point, i.e.\ to an abstract state. The level of abstraction can be adjusted by determining the number of fixed points of the modern Hopfield network. Furthermore, \textit{contrastive abstraction learning} does not require rewards and facilitates efficient reinforcement learning for a wide range of downstream tasks. Our experiments demonstrate the effectiveness of contrastive abstraction learning for reinforcement learning.