Information-Theoretic Methods for Planning and Learning in Partially Observable Markov Decision Processes

TL;DR

This paper develops information-theoretic methods to optimize planning and learning in partially observable Markov decision processes, accounting for agents' intrinsic information-processing constraints and analyzing their effects on agent behavior and convergence.

Contribution

It formulates a framework for optimizing agents under information constraints, analyzes convergence issues, and provides solutions for linear-Gaussian cases and learning scenarios.

Findings

Optimal solutions for constrained agents are derived.

Bifurcation structures affect convergence properties.

Sample-based learning methods are explored for unknown dynamics.

Abstract

Bounded agents are limited by intrinsic constraints on their ability to process information that is available in their sensors and memory and choose actions and memory updates. In this dissertation, we model these constraints as information-rate constraints on communication channels connecting these various internal components of the agent. We make four major contributions detailed below and many smaller contributions detailed in each section. First, we formulate the problem of optimizing the agent under both extrinsic and intrinsic constraints and develop the main tools for solving it. Second, we identify another reason for the challenging convergence properties of the optimization algorithm, which is the bifurcation structure of the update operator near phase transitions. Third, we study the special case of linear-Gaussian dynamics and quadratic cost (LQG), where the optimal solution has a particularly simple and solvable form. Fourth, we explore the learning task, where the model of the world dynamics is unknown and sample-based updates are used instead.