Evaluating Self-Correcting Vision Agents Through Quantitative and Qualitative Metrics

TL;DR

This paper introduces a diagnostic benchmark for Vision-Language Agents, analyzing their self-correction capabilities and identifying key reasoning bottlenecks like semantic drift to improve multimodal AI reliability.

Contribution

It presents a new micro-benchmark that separates task success from correction success, quantifies diminishing returns of self-correction, and identifies semantic drift as a major failure factor.

Findings

Task success rate is 62% while correction success rate is 25-33%.

Correction attempts saturate after three retries.

Semantic drift accounts for about 28% of failures.

Abstract

Recent progress in multimodal foundation models has enabled Vision-Language Agents (VLAs) to decompose complex visual tasks into executable tool-based plans. While recent benchmarks have begun to evaluate iterative self-correction, its quantitative limits and dominant reasoning bottlenecks remain poorly characterized. This work introduces a Diagnostic Micro-Benchmark. Our analysis decouples Task Success Rate (TSR = 62 percent) from Correction Success Rate (CSR = 25 to 33 percent), revealing that initial competence does not predict repair ability. We explicitly quantify the diminishing returns of correction, which saturates after three retries. Our Failure Taxonomy reveals a frequent factor is Semantic Drift (about 28 percent of failures), a loss of contextual state. By isolating this reasoning bottleneck, this benchmark defines a reproducible framework toward stateful, trustworthy multimodal agents.