Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models

TL;DR

This paper introduces ERL-VLM, a novel rating-based reinforcement learning method that leverages large vision-language models for AI-generated feedback, improving reward learning efficiency and stability without extensive human input.

Contribution

ERL-VLM enhances rating-based RL by using absolute ratings from VLMs and addresses data imbalance and noise issues, advancing AI-driven reward learning.

Findings

ERL-VLM outperforms existing VLM-based reward methods in various tasks.

The method improves sample efficiency and stability in reward learning.

AI feedback can effectively scale RL with minimal human supervision.

Abstract

Designing effective reward functions remains a fundamental challenge in reinforcement learning (RL), as it often requires extensive human effort and domain expertise. While RL from human feedback has been successful in aligning agents with human intent, acquiring high-quality feedback is costly and labor-intensive, limiting its scalability. Recent advancements in foundation models present a promising alternative--leveraging AI-generated feedback to reduce reliance on human supervision in reward learning. Building on this paradigm, we introduce ERL-VLM, an enhanced rating-based RL method that effectively learns reward functions from AI feedback. Unlike prior methods that rely on pairwise comparisons, ERL-VLM queries large vision-language models (VLMs) for absolute ratings of individual trajectories, enabling more expressive feedback and improved sample efficiency. Additionally, we propose key enhancements to rating-based RL, addressing instability issues caused by data imbalance and noisy labels. Through extensive experiments across both low-level and high-level control tasks, we demonstrate that ERL-VLM significantly outperforms existing VLM-based reward generation methods. Our results demonstrate the potential of AI feedback for scaling RL with minimal human intervention, paving the way for more autonomous and efficient reward learning.