Semantic HELM: A Human-Readable Memory for Reinforcement Learning

TL;DR

This paper introduces Semantic HELM, a human-readable memory system for reinforcement learning agents that uses language tokens to represent past events, improving interpretability and efficiency in memory-dependent tasks.

Contribution

It proposes a novel memory mechanism combining CLIP and pretrained language models to create interpretable, human-readable memory representations for reinforcement learning agents.

Findings

Outperforms baselines on memory-dependent tasks

Converges faster on challenging recognition tasks

Enhances interpretability and troubleshooting of agents

Abstract

Reinforcement learning agents deployed in the real world often have to cope with partially observable environments. Therefore, most agents employ memory mechanisms to approximate the state of the environment. Recently, there have been impressive success stories in mastering partially observable environments, mostly in the realm of computer games like Dota 2, StarCraft II, or MineCraft. However, existing methods lack interpretability in the sense that it is not comprehensible for humans what the agent stores in its memory. In this regard, we propose a novel memory mechanism that represents past events in human language. Our method uses CLIP to associate visual inputs with language tokens. Then we feed these tokens to a pretrained language model that serves the agent as memory and provides it with a coherent and human-readable representation of the past. We train our memory mechanism on a set of partially observable environments and find that it excels on tasks that require a memory component, while mostly attaining performance on-par with strong baselines on tasks that do not. On a challenging continuous recognition task, where memorizing the past is crucial, our memory mechanism converges two orders of magnitude faster than prior methods. Since our memory mechanism is human-readable, we can peek at an agent's memory and check whether crucial pieces of information have been stored. This significantly enhances troubleshooting and paves the way toward more interpretable agents.