FoMo Rewards: Can we cast foundation models as reward functions?

TL;DR

This paper investigates using foundation models as reward functions in reinforcement learning by combining vision and language models to evaluate task adherence, enabling open-ended interactive agents.

Contribution

It introduces a pipeline that uses existing vision and language models to generate reward signals, demonstrating a new way to leverage foundation models for reinforcement learning.

Findings

Likelihood functions correlate with desired behaviors

High likelihood for correct policies, lower for incorrect ones

Potential for open-ended interactive agent design

Abstract

We explore the viability of casting foundation models as generic reward functions for reinforcement learning. To this end, we propose a simple pipeline that interfaces an off-the-shelf vision model with a large language model. Specifically, given a trajectory of observations, we infer the likelihood of an instruction describing the task that the user wants an agent to perform. We show that this generic likelihood function exhibits the characteristics ideally expected from a reward function: it associates high values with the desired behaviour and lower values for several similar, but incorrect policies. Overall, our work opens the possibility of designing open-ended agents for interactive tasks via foundation models.