SAGE: Generating Symbolic Goals for Myopic Models in Deep Reinforcement Learning

TL;DR

SAGE is a novel algorithm that combines symbolic planning and neural learning to effectively utilize incomplete models in deep reinforcement learning, improving performance in complex, sparse reward environments.

Contribution

It introduces a new hybrid approach that leverages incomplete models without restrictive assumptions, enhancing sample efficiency and applicability in complex domains.

Findings

Outperforms existing methods on taxi world and Minecraft tasks.

Effectively exploits incomplete models for better planning and learning.

Demonstrates improved sample efficiency in sparse reward environments.

Abstract

Model-based reinforcement learning algorithms are typically more sample efficient than their model-free counterparts, especially in sparse reward problems. Unfortunately, many interesting domains are too complex to specify the complete models required by traditional model-based approaches. Learning a model takes a large number of environment samples, and may not capture critical information if the environment is hard to explore. If we could specify an incomplete model and allow the agent to learn how best to use it, we could take advantage of our partial understanding of many domains. Existing hybrid planning and learning systems which address this problem often impose highly restrictive assumptions on the sorts of models which can be used, limiting their applicability to a wide range of domains. In this work we propose SAGE, an algorithm combining learning and planning to exploit a previously unusable class of incomplete models. This combines the strengths of symbolic planning and neural learning approaches in a novel way that outperforms competing methods on variations of taxi world and Minecraft.