Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making

TL;DR

This paper explores Pavlovian signalling as a mechanism for adaptive communication between learning agents, demonstrating its effectiveness in a novel decision-making domain and analyzing its impact on coordination and timing.

Contribution

It introduces Pavlovian signalling as a bridge between fixed signals and adaptive communication, showing how to build it from prediction learning with minimal constraints.

Findings

Pavlovian signalling accelerates learning in agent interactions.

Temporal representations influence coordination but not the speed of learning.

Temporal aliasing affects human-agent and agent-agent interactions differently.

Abstract

In this paper, we contribute a multi-faceted study into Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent. Signalling is intimately connected to time and timing. In service of generating and receiving signals, humans and other animals are known to represent time, determine time since past events, predict the time until a future stimulus, and both recognize and generate patterns that unfold in time. We investigate how different temporal processes impact coordination and signalling between learning agents by introducing a partially observable decision-making domain we call the Frost Hollow. In this domain, a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that works to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: machine agents interacting in a seven-state linear walk, and human-machine interaction in a virtual-reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning between two agents. We further show how to computationally build this adaptive signalling process out of a fixed signalling process, characterized by fast continual prediction learning and minimal constraints on the nature of the agent receiving signals. Our results therefore suggest an actionable, constructivist path towards communication learning between reinforcement learning agents.