Relate to Predict: Towards Task-Independent Knowledge Representations for Reinforcement Learning

TL;DR

This paper introduces an explicit object-centered knowledge separation method in reinforcement learning, utilizing semantic modules to improve interpretability, generalization, and learning efficiency across puzzle-like tasks.

Contribution

It proposes a novel semantic module for explicit object-centered knowledge separation, enhancing RL models with better interpretability and generalization capabilities.

Findings

Explicit knowledge separation improves learning speed

Semantic representations enhance generalization

Method increases interpretability of RL models

Abstract

Reinforcement Learning (RL) can enable agents to learn complex tasks. However, it is difficult to interpret the knowledge and reuse it across tasks. Inductive biases can address such issues by explicitly providing generic yet useful decomposition that is otherwise difficult or expensive to learn implicitly. For example, object-centered approaches decompose a high dimensional observation into individual objects. Expanding on this, we utilize an inductive bias for explicit object-centered knowledge separation that provides further decomposition into semantic representations and dynamics knowledge. For this, we introduce a semantic module that predicts an objects' semantic state based on its context. The resulting affordance-like object state can then be used to enrich perceptual object representations. With a minimal setup and an environment that enables puzzle-like tasks, we demonstrate the feasibility and benefits of this approach. Specifically, we compare three different methods of integrating semantic representations into a model-based RL architecture. Our experiments show that the degree of explicitness in knowledge separation correlates with faster learning, better accuracy, better generalization, and better interpretability.