Enhancing Agentic Autonomous Scientific Discovery with Vision-Language Model Capabilities

TL;DR

This paper demonstrates that integrating vision-language models into multi-agent systems significantly enhances autonomous scientific discovery by enabling real-time evaluation, correction, and adaptation in complex data analysis tasks.

Contribution

It introduces a novel framework where VLMs serve as judges for scientific figures, improving autonomous data analysis and discovery without human intervention.

Findings

VLM-guided agents outperform baselines on discovery benchmarks.

Agents recover from faulty reasoning using VLM evaluations.

The system provides interpretable reasoning traces.

Abstract

We show that multi-agent systems guided by vision-language models (VLMs) improve end-to-end autonomous scientific discovery. By treating plots as verifiable checkpoints, a VLM-as-a-judge evaluates figures against dynamically generated domain-specific rubrics, enabling agents to correct their own errors and steer exploratory data analysis in real-time. Case studies in cosmology and astrochemistry demonstrate recovery from faulty reasoning paths and adaptation to new datasets without human intervention. On a 10-task benchmark for data-driven discovery, VLM-augmented systems achieve pass at 1 scores of 0.7-0.8, compared to 0.2-0.3 for code-only and 0.4-0.5 for code-and-text baselines, while also providing auditable reasoning traces that improve interpretability. Code available here: https://github.com/CMBAgents/cmbagent