VIRAL: Vision-grounded Integration for Reward design And Learning

TL;DR

VIRAL is a novel pipeline that uses multi-modal large language models to generate and refine reward functions for reinforcement learning, improving alignment with human intent and accelerating learning in various environments.

Contribution

It introduces VIRAL, a new method for autonomous reward function creation and refinement using multi-modal LLMs, enhancing AI alignment and learning efficiency.

Findings

VIRAL accelerates learning of new behaviors in Gym environments.

It improves alignment with user intent through interactive reward refinement.

VIRAL demonstrates effective use of multi-modal LLMs for reward generation.

Abstract

The alignment between humans and machines is a critical challenge in artificial intelligence today. Reinforcement learning, which aims to maximize a reward function, is particularly vulnerable to the risks associated with poorly designed reward functions. Recent advancements has shown that Large Language Models (LLMs) for reward generation can outperform human performance in this context. We introduce VIRAL, a pipeline for generating and refining reward functions through the use of multi-modal LLMs. VIRAL autonomously creates and interactively improves reward functions based on a given environment and a goal prompt or annotated image. The refinement process can incorporate human feedback or be guided by a description generated by a video LLM, which explains the agent's policy in video form. We evaluated VIRAL in five Gymnasium environments, demonstrating that it accelerates the learning of new behaviors while ensuring improved alignment with user intent. The source-code and demo video are available at: https://github.com/VIRAL-UCBL1/VIRAL and https://youtu.be/Hqo82CxVT38.