Will it Blend? Composing Value Functions in Reinforcement Learning

TL;DR

This paper introduces a principled method for composing value functions in reinforcement learning, enabling agents to solve new tasks by combining existing skills without additional training.

Contribution

It provides a theoretical framework for optimal value function composition in entropy-regularised RL and extends it to standard RL, demonstrated through a video game example.

Findings

Agents can solve new tasks by composing existing policies.

The composition method is optimal in entropy-regularised RL.

Demonstrated effectiveness in a video game environment.

Abstract

An important property for lifelong-learning agents is the ability to combine existing skills to solve unseen tasks. In general, however, it is unclear how to compose skills in a principled way. We provide a "recipe" for optimal value function composition in entropy-regularised reinforcement learning (RL) and then extend this to the standard RL setting. Composition is demonstrated in a video game environment, where an agent with an existing library of policies is able to solve new tasks without the need for further learning.