VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning

TL;DR

VISCO introduces a comprehensive benchmark for evaluating fine-grained critique and correction in vision-language models, revealing current limitations and proposing a strategy that enhances self-improvement in visual reasoning tasks.

Contribution

This work presents the first benchmark for dense critique and correction in LVLMs, analyzes their capabilities, identifies common failure patterns, and proposes the LookBack strategy to improve performance.

Findings

Human critiques significantly improve correction performance.

Model-generated critiques are less effective and sometimes harmful.

LookBack strategy enhances critique and correction by up to 13.5%.

Abstract

The ability of large vision-language models (LVLMs) to critique and correct their reasoning is an essential building block towards their self-improvement. However, a systematic analysis of such capabilities in LVLMs is still lacking. We propose VISCO, the first benchmark to extensively analyze the fine-grained critique and correction capabilities of LVLMs. Compared to existing work that uses a single scalar value to critique the entire reasoning [4], VISCO features dense and fine-grained critique, requiring LVLMs to evaluate the correctness of each step in the chain-of-thought and provide natural language explanations to support their judgments. Extensive evaluation of 24 LVLMs demonstrates that human-written critiques significantly enhance the performance after correction, showcasing the potential of the self-improvement strategy. However, the model-generated critiques are less helpful and sometimes detrimental to the performance, suggesting that critique is the crucial bottleneck. We identified three common patterns in critique failures: failure to critique visual perception, reluctance to "say no", and exaggerated assumption of error propagation. To address these issues, we propose an effective LookBack strategy that revisits the image to verify each piece of information in the initial reasoning. LookBack significantly improves critique and correction performance by up to 13.5%.