VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents

TL;DR

VLGOR introduces a framework that combines visual and language knowledge to generate imaginary environment interactions, enhancing offline reinforcement learning for better generalization to unseen tasks.

Contribution

The paper presents a novel method that integrates visual-language models with offline RL, enabling the generation of diverse, coherent rollouts for improved task generalization.

Findings

Achieves over 24% higher success rate on robotic manipulation benchmarks.

Effectively generates diverse and plausible environment rollouts.

Enhances agent performance on unseen tasks with novel policies.

Abstract

Combining Large Language Models (LLMs) with Reinforcement Learning (RL) enables agents to interpret language instructions more effectively for task execution. However, LLMs typically lack direct perception of the physical environment, which limits their understanding of environmental dynamics and their ability to generalize to unseen tasks. To address this limitation, we propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts, thereby enriching the interaction data. The core premise of VLGOR is to fine-tune a vision-language model to predict future states and actions conditioned on an initial visual observation and high-level instructions, ensuring that the generated rollouts remain temporally coherent and spatially plausible. Furthermore, we employ counterfactual prompts to produce more diverse rollouts for offline RL training, enabling the agent to acquire knowledge that facilitates following language instructions while grounding in environments based on visual cues. Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.