Learning telic-controllable state representations

TL;DR

This paper introduces a novel framework for learning telic-controllable state representations in reinforcement learning, emphasizing the interplay between goal-directed states and policy complexity to improve goal flexibility and cognitive efficiency.

Contribution

It presents a new computational framework and algorithm for coupling descriptive and prescriptive state aspects through telic states, highlighting the role of deliberate ignorance.

Findings

The framework effectively balances goal flexibility and policy complexity.

The algorithm successfully learns telic-controllable state representations in simulated navigation.

Deliberate ignorance enhances the learning of goal-oriented state representations.

Abstract

Computational models of purposeful behavior comprise both descriptive and prescriptive aspects, used respectively to ascertain and evaluate situations in the world. In reinforcement learning, prescriptive reward functions are assumed to depend on predefined and fixed descriptive state representations. Alternatively, these two aspects may emerge interdependently: goals can shape the acquired state representations and vice versa. Here, we present a computational framework for state representation learning in bounded agents, where descriptive and prescriptive aspects are coupled through the notion of goal-directed, or telic, states. We introduce the concept of telic-controllability to characterize the tradeoff between the granularity of a telic state representation and the policy complexity required to reach all telic states. We propose an algorithm for learning telic-controllable state representations, illustrating it using a simulated navigation task. Our framework highlights the role of deliberate ignorance -- knowing what to ignore -- for learning state representations that balance goal flexibility and cognitive complexity.