COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control

TL;DR

COVR introduces a collaborative framework that mutually enhances vision-language models and reinforcement learning policies through RL-generated data, improving sample efficiency and performance in visual control tasks.

Contribution

The paper proposes a novel collaborative optimization framework that jointly fine-tunes VLMs and RL policies, incorporating new modules for efficient training and mutual enhancement.

Findings

COVR outperforms existing methods on various visual control benchmarks.

The proposed modules improve training stability and exploration efficiency.

Mutual enhancement leads to better semantic reasoning and policy learning.

Abstract

Visual reinforcement learning (RL) suffers from poor sample efficiency due to high-dimensional observations in complex tasks. While existing works have shown that vision-language models (VLMs) can assist RL, they often focus on knowledge distillation from the VLM to RL, overlooking the potential of RL-generated interaction data to enhance the VLM. To address this, we propose COVR, a collaborative optimization framework that enables the mutual enhancement of the VLM and RL policies. Specifically, COVR fine-tunes the VLM with RL-generated data to enhance the semantic reasoning ability consistent with the target task, and uses the enhanced VLM to further guide policy learning via action priors. To improve fine-tuning efficiency, we introduce two key modules: (1) an Exploration-Driven Dynamic Filter module that preserves valuable exploration samples using adaptive thresholds based on the degree of exploration, and (2) a Return-Aware Adaptive Loss Weight module that improves the stability of training by quantifying the inconsistency of sampling actions via return signals of RL. We further design a progressive fine-tuning strategy to reduce resource consumption. Extensive experiments show that COVR achieves strong performance across various challenging visual control tasks.