World Value Functions: Knowledge Representation for Multitask Reinforcement Learning

TL;DR

This paper introduces world value functions (WVFs), a novel knowledge representation enabling agents to learn, infer, and reuse solutions for multiple tasks in a given environment, supporting zero-shot generalization and planning.

Contribution

The work proposes WVFs as a unified framework for representing and learning knowledge that generalizes across tasks, allowing inference of policies and values for new tasks and enabling zero-shot skill composition.

Findings

WVFs can infer transition functions from learned values.

Pretrained WVFs enable direct inference of policies for new tasks.

Agents can solve combined tasks zero-shot using WVFs.

Abstract

An open problem in artificial intelligence is how to learn and represent knowledge that is sufficient for a general agent that needs to solve multiple tasks in a given world. In this work we propose world value functions (WVFs), which are a type of general value function with mastery of the world - they represent not only how to solve a given task, but also how to solve any other goal-reaching task. To achieve this, we equip the agent with an internal goal space defined as all the world states where it experiences a terminal transition - a task outcome. The agent can then modify task rewards to define its own reward function, which provably drives it to learn how to achieve all achievable internal goals, and the value of doing so in the current task. We demonstrate a number of benefits of WVFs. When the agent's internal goal space is the entire state space, we demonstrate that the transition function can be inferred from the learned WVF, which allows the agent to plan using learned value functions. Additionally, we show that for tasks in the same world, a pretrained agent that has learned any WVF can then infer the policy and value function for any new task directly from its rewards. Finally, an important property for long-lived agents is the ability to reuse existing knowledge to solve new tasks. Using WVFs as the knowledge representation for learned tasks, we show that an agent is able to solve their logical combination zero-shot, resulting in a combinatorially increasing number of skills throughout their lifetime.