Action and Perception as Divergence Minimization

TL;DR

This paper introduces the Action Perception Divergence (APD) framework, categorizing objectives for embodied agents from narrow rewards to general information-maximizing goals, unifying many unsupervised learning approaches under a single principle.

Contribution

The paper proposes the APD framework, providing a unified perspective on diverse objectives for agents, linking reinforcement learning, representation learning, and intrinsic motivation.

Findings

APD categorizes objectives from narrow rewards to general information maximization.

Agents using APD principles can explore and adapt without explicit task rewards.

The framework unifies various unsupervised learning objectives under a common principle.

Abstract

To learn directed behaviors in complex environments, intelligent agents need to optimize objective functions. Various objectives are known for designing artificial agents, including task rewards and intrinsic motivation. However, it is unclear how the known objectives relate to each other, which objectives remain yet to be discovered, and which objectives better describe the behavior of humans. We introduce the Action Perception Divergence (APD), an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general objectives. While the narrow objectives correspond to domain-specific rewards as typical in reinforcement learning, the general objectives maximize information with the environment through latent variable models of input sequences. Intuitively, these agents use perception to align their beliefs with the world and use actions to align the world with their beliefs. They infer representations that are informative of past inputs, explore future inputs that are informative of their representations, and select actions or skills that maximally influence future inputs. This explains a wide range of unsupervised objectives from a single principle, including representation learning, information gain, empowerment, and skill discovery. Our findings suggest leveraging powerful world models for unsupervised exploration as a path toward highly adaptive agents that seek out large niches in their environments, rendering task rewards optional.