Can Argus Judge Them All? Comparing VLMs Across Domains

TL;DR

This paper benchmarks major vision-language models across diverse tasks and datasets, revealing their strengths, weaknesses, and trade-offs in generalization, specialization, and robustness to guide future development and deployment.

Contribution

It introduces a comprehensive benchmarking framework including a novel Cross-Dataset Consistency metric to evaluate VLMs across multiple domains and tasks.

Findings

CLIP shows strongest generalization (CDC: 0.92)

BLIP excels on curated data

LXMERT leads in structured reasoning

Abstract

Vision-Language Models (VLMs) are advancing multimodal AI, yet their performance consistency across tasks is underexamined. We benchmark CLIP, BLIP, and LXMERT across diverse datasets spanning retrieval, captioning, and reasoning. Our evaluation includes task accuracy, generation quality, efficiency, and a novel Cross-Dataset Consistency (CDC) metric. CLIP shows strongest generalization (CDC: 0.92), BLIP excels on curated data, and LXMERT leads in structured reasoning. These results expose trade-offs between generalization and specialization, informing industrial deployment of VLMs and guiding development toward robust, task-flexible architectures.