How to Learn and Represent Abstractions: An Investigation using Symbolic Alchemy

TL;DR

This paper introduces Symbolic Alchemy, a meta-learning environment designed to study how deep reinforcement learning agents learn and represent abstractions, with insights connected to neuroscience and potential for understanding brain representations.

Contribution

The work pioneers using Symbolic Alchemy to analyze abstraction learning in deep-RL agents and explores the neural-like representations of abstract variables.

Findings

Agents learn various types of abstractions in Alchemy.

Behavioral and introspective analyses reveal neural-like representations.

Connections drawn between agent representations and neuroscience of abstraction.

Abstract

Alchemy is a new meta-learning environment rich enough to contain interesting abstractions, yet simple enough to make fine-grained analysis tractable. Further, Alchemy provides an optional symbolic interface that enables meta-RL research without a large compute budget. In this work, we take the first steps toward using Symbolic Alchemy to identify design choices that enable deep-RL agents to learn various types of abstraction. Then, using a variety of behavioral and introspective analyses we investigate how our trained agents use and represent abstract task variables, and find intriguing connections to the neuroscience of abstraction. We conclude by discussing the next steps for using meta-RL and Alchemy to better understand the representation of abstract variables in the brain.