An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Models

TL;DR

MultiNet is an open-source benchmark suite that evaluates and adapts multimodal action models across vision, language, and action tasks, fostering progress in general-purpose agentic systems.

Contribution

It introduces a comprehensive benchmark, software ecosystem, and standardized evaluation protocols for multimodal models, with a large composite dataset for diverse tasks.

Findings

Used in research to identify limitations of VLA generalization.

Provides standardized evaluation protocols and open-source tools.

Enables rigorous assessment and adaptation of multimodal models.

Abstract

Recent innovations in multimodal action models represent a promising direction for developing general-purpose agentic systems, combining visual understanding, language comprehension, and action generation. We introduce MultiNet - a novel, fully open-source benchmark and surrounding software ecosystem designed to rigorously evaluate and adapt models across vision, language, and action domains. We establish standardized evaluation protocols for assessing vision-language models (VLMs) and vision-language-action models (VLAs), and provide open source software to download relevant data, models, and evaluations. Additionally, we provide a composite dataset with over 1.3 trillion tokens of image captioning, visual question answering, commonsense reasoning, robotic control, digital game-play, simulated locomotion/manipulation, and many more tasks. The MultiNet benchmark, framework, toolkit, and evaluation harness have been used in downstream research on the limitations of VLA generalization.