VLM-Guided Experience Replay

TL;DR

This paper introduces a novel method that uses pre-trained Vision-Language Models to prioritize experiences in reinforcement learning replay buffers, significantly improving sample efficiency and success rates across various domains.

Contribution

It proposes leveraging frozen VLMs to guide experience replay prioritization, a novel approach that enhances RL performance without additional training of the VLM.

Findings

Achieved 11-52% higher success rates in various tasks.

Improved sample efficiency by 19-45%.

Effective across game-playing and robotics scenarios.

Abstract

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM (requiring no fine-tuning) as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches. https://esharony.me/projects/vlm-rb/