Formalising the intentional stance 2: a coinductive approach

TL;DR

This paper formalizes the concept of goal-directed behavior in stochastic systems using coinductive definitions, extending decision theory to bounded rationality and providing proofs for properties of optimal policies.

Contribution

It introduces a coinductive framework for modeling rational and bounded-rational policies in stochastic processes, with formal proofs and decision-theoretic characterizations.

Findings

Optimal policies obey Bellman's principle with Bayesian updates.

Deterministic policies are uniquely optimal for some teleo-environments.

Bounded rationality may violate classical representation theorems.

Abstract

Given a stochastic process with inputs and outputs, how might its behaviour be related to pursuit of a goal? We model this using 'transducers', objects that capture only the external behaviour of a system and not its internal state. A companion paper summarises our results for cognitive scientists; the current paper gives formal definitions and proofs. To formalise the concept of a system that behaves as if it were pursuing a goal, we consider what happens when a transducer (a 'policy') is coupled to another transducer that comes equipped with a success condition (a 'teleo-environment'). An optimal policy is identified with a transducer that behaves as if it were perfectly rational in the pursuit of a goal; our framework also allows us to model constrained rationality. Optimal policies obey a version of Bellman's principle: a policy that's optimal in one time step will again be optimal in the next time step, but with respect to a different teleo-environment (obtained from the original one by a modified version of Bayesian filtering). This property sometimes also applies to the bounded-rational case; we give a sufficient condition. A policy is deterministic if and only if there exists a teleo-environment for which it is uniquely optimal among the set of all policies; we relate this to classical representation theorems from decision theory. This result need not hold in the bounded-rational case; we give an example related to the absent-minded driver problem. The formalism is defined using coinduction, following the style proposed by Czajka.