MA-VLCM: A Vision Language Critic Model for Value Estimation of Policies in Multi-Agent Team Settings

TL;DR

This paper introduces MA-VLCM, a framework that leverages pretrained vision-language models as critics in multi-agent reinforcement learning, enhancing sample efficiency and enabling deployment on resource-limited robots.

Contribution

The paper proposes replacing learned critics with pretrained vision-language models fine-tuned for multi-agent value estimation, improving efficiency and generalization.

Findings

Achieves good zero-shot return estimation in multi-agent scenarios.

Significantly improves sample efficiency in policy training.

Enables deployment on resource-constrained robotic systems.

Abstract

Multi-agent reinforcement learning (MARL) commonly relies on a centralized critic to estimate the value function. However, learning such a critic from scratch is highly sample-inefficient and often lacks generalization across environments. At the same time, large vision-language-action models (VLAs) trained on internet-scale data exhibit strong multimodal reasoning and zero-shot generalization capabilities, yet directly deploying them for robotic execution remains computationally prohibitive, particularly in heterogeneous multi-robot systems with diverse embodiments and resource constraints. To address these challenges, we propose Multi-Agent Vision-Language-Critic Models (MA-VLCM), a framework that replaces the learned centralized critic in MARL with a pretrained vision-language model fine-tuned to evaluate multi-agent behavior. MA-VLCM acts as a centralized critic conditioned on natural language task descriptions, visual trajectory observations, and structured multi-agent state information. By eliminating critic learning during policy optimization, our approach significantly improves sample efficiency while producing compact execution policies suitable for deployment on resource-constrained robots. Results show good zero-shot return estimation on models with differing VLM backbones on in-distribution and out-of-distribution scenarios in multi-agent team settings