The Thing That We Tried Didn't Work Very Well : Deictic Representation in Reinforcement Learning

TL;DR

This paper investigates deictic representations in reinforcement learning within a blocks-world domain, finding that they perform worse than propositional representations and discussing potential reasons and future strategies.

Contribution

It provides an empirical comparison of deictic and propositional representations, revealing the limitations of deictic approaches in this context.

Findings

Deictic representations worsened learning performance

Propositional representations performed better in the experiments

Discussion of causes and strategies for improvement

Abstract

Most reinforcement learning methods operate on propositional representations of the world state. Such representations are often intractably large and generalize poorly. Using a deictic representation is believed to be a viable alternative: they promise generalization while allowing the use of existing reinforcement-learning methods. Yet, there are few experiments on learning with deictic representations reported in the literature. In this paper we explore the effectiveness of two forms of deictic representation and a na\"{i}ve propositional representation in a simple blocks-world domain. We find, empirically, that the deictic representations actually worsen learning performance. We conclude with a discussion of possible causes of these results and strategies for more effective learning in domains with objects.