Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines

TL;DR

This paper introduces Multitask Preplay, a novel algorithm that enables humans and machines to learn and generalize across multiple tasks by simulating unpursued tasks based on experience, improving adaptability.

Contribution

The paper formalizes Multitask Preplay, demonstrating its effectiveness in predicting human generalization and enhancing artificial agent performance in complex environments.

Findings

Multitask Preplay better predicts human generalization in grid-world tasks.

The approach generalizes successfully to the Craftax Minecraft environment.

Artificial agents using Multitask Preplay transfer skills to new Craftax worlds.

Abstract

Humans can pursue a near-infinite variety of tasks, but typically can only pursue a small number at the same time. We hypothesize that humans leverage experience on one task to preemptively learn solutions to other tasks that were accessible but not pursued. We formalize this idea as Multitask Preplay, a novel algorithm that replays experience on one task as the starting point for "preplay" -- counterfactual simulation of an accessible but unpursued task. Preplay is used to learn a predictive representation that can support fast, adaptive task performance later on. We first show that, compared to traditional planning and predictive representation methods, multitask preplay better predicts how humans generalize to tasks that were accessible but not pursued in a small grid-world, even when people didn't know they would need to generalize to these tasks. We then show these predictions generalize to Craftax, a partially observable 2D Minecraft environment. Finally, we show that Multitask Preplay enables artificial agents to learn behaviors that transfer to novel Craftax worlds sharing task co-occurrence structure. These findings demonstrate that Multitask Preplay is a scalable theory of how humans counterfactually learn and generalize across multiple tasks; endowing artificial agents with the same capacity can significantly improve their performance in challenging multitask environments.