Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments

TL;DR

This paper introduces MultiNet v0.2, a benchmark for evaluating vision-language-action models' zero-shot generalization in procedurally generated, out-of-distribution environments, revealing key limitations and strengths of current models.

Contribution

The paper presents a new comprehensive benchmark, MultiNet v0.2, for assessing the generalization of VLA models in OOD environments, along with an analysis of their performance and factors affecting it.

Findings

All models show limited zero-shot OOD generalization.

VLAs outperform other models due to architecture.

Prompt engineering significantly affects model performance.

Abstract

Vision-language-action (VLA) models represent an important step toward general-purpose robotic systems by integrating visual perception, language understanding, and action execution. However, systematic evaluation of these models, particularly their zero-shot generalization capabilities in procedurally out-of-distribution (OOD) environments, remains limited. In this paper, we introduce MultiNet v0.2, a comprehensive benchmark designed to evaluate and analyze the generalization performance of state-of-the-art VLMs and VLAs - including GPT-4o, GPT-4.1, OpenVLA, Pi0 Base, and Pi0 FAST - on diverse procedural tasks from the Procgen benchmark. Our analysis reveals several critical insights: (1) all evaluated models exhibit significant limitations in zero-shot generalization to OOD tasks, with performance heavily influenced by factors such as action representation and task complexity; (2) VLAs generally outperforms other models due to their robust architectural design; and (3) VLM variants demonstrate substantial improvements when constrained appropriately, highlighting the sensitivity of model performance to precise prompt engineering. We release our benchmark, evaluation framework, and findings to enable the assessment of future VLA models and identify critical areas for improvement in their application to out-of-distribution digital tasks.