Grounded Intuition of GPT-Vision's Abilities with Scientific Images

TL;DR

This paper introduces a qualitative framework to evaluate GPT-Vision's abilities with scientific images, revealing its sensitivities and aiding researchers in understanding its capabilities and limitations.

Contribution

It develops a grounded, example-driven qualitative evaluation method for GPT-Vision, moving beyond traditional benchmarks to better understand model behavior.

Findings

GPT-Vision is sensitive to prompting and counterfactual text.

It effectively captures relative spatial relationships in images.

The framework helps develop grounded intuition of model capabilities.

Abstract

GPT-Vision has impressed us on a range of vision-language tasks, but it comes with the familiar new challenge: we have little idea of its capabilities and limitations. In our study, we formalize a process that many have instinctively been trying already to develop "grounded intuition" of this new model. Inspired by the recent movement away from benchmarking in favor of example-driven qualitative evaluation, we draw upon grounded theory and thematic analysis in social science and human-computer interaction to establish a rigorous framework for qualitative evaluation in natural language processing. We use our technique to examine alt text generation for scientific figures, finding that GPT-Vision is particularly sensitive to prompting, counterfactual text in images, and relative spatial relationships. Our method and analysis aim to help researchers ramp up their own grounded intuitions of new models while exposing how GPT-Vision can be applied to make information more accessible.