How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model?

TL;DR

This paper introduces a methodology to identify preferred image distributions for black-box vision-language models, applies it to 3D object rendering, and presents a new CAD-specific dataset for visual question answering to advance complex visual reasoning.

Contribution

It proposes a novel, generalizable method for determining preferred image inputs for black-box VLMs and introduces CAD-VQA, a new dataset for CAD-related visual reasoning tasks.

Findings

Effective in identifying preferred image distributions across domains.

Significantly improves explanation quality with human feedback.

Establishes baseline performance on CAD-VQA dataset.

Abstract

Large foundation models have revolutionized the field, yet challenges remain in optimizing multi-modal models for specialized visual tasks. We propose a novel, generalizable methodology to identify preferred image distributions for black-box Vision-Language Models (VLMs) by measuring output consistency across varied input prompts. Applying this to different rendering types of 3D objects, we demonstrate its efficacy across various domains requiring precise interpretation of complex structures, with a focus on Computer-Aided Design (CAD) as an exemplar field. We further refine VLM outputs using in-context learning with human feedback, significantly enhancing explanation quality. To address the lack of benchmarks in specialized domains, we introduce CAD-VQA, a new dataset for evaluating VLMs on CAD-related visual question answering tasks. Our evaluation of state-of-the-art VLMs on CAD-VQA establishes baseline performance levels, providing a framework for advancing VLM capabilities in complex visual reasoning tasks across various fields requiring expert-level visual interpretation. We release the dataset and evaluation codes at \url{https://github.com/asgsaeid/cad_vqa}.