InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction

TL;DR

InfantAgent-Next is a multimodal generalist agent capable of interacting with computers across text, images, audio, and video, integrating tools and vision models in a modular architecture to solve diverse tasks.

Contribution

The paper presents a novel modular multimodal agent architecture that combines tool-based and vision models, enabling versatile computer interaction and evaluation on various benchmarks.

Findings

Achieves 7.27% accuracy on OSWorld benchmark.

Successfully evaluates on both vision-based and tool-intensive benchmarks.

Open-sourced code and evaluation scripts at GitHub.

Abstract

This paper introduces \textsc{InfantAgent-Next}, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video. Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench). Specifically, we achieve $\mathbf{7.27\%}$ accuracy on OSWorld, higher than Claude-Computer-Use. Codes and evaluation scripts are open-sourced at https://github.com/bin123apple/InfantAgent.