VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks

TL;DR

VL-GLUE is a comprehensive benchmark with over 100,000 samples across seven tasks, designed to evaluate and advance visuo-linguistic reasoning in AI systems by challenging current models with diverse, real-world multimodal data.

Contribution

This paper introduces VL-GLUE, a new large-scale benchmark for visuo-linguistic reasoning, highlighting its diversity and difficulty for existing models.

Findings

Current models struggle with VL-GLUE tasks.

VL-GLUE covers diverse image types and domain-specific texts.

Benchmark encourages development of more robust visuo-linguistic AI systems.

Abstract

Deriving inference from heterogeneous inputs (such as images, text, and audio) is an important skill for humans to perform day-to-day tasks. A similar ability is desirable for the development of advanced Artificial Intelligence (AI) systems. While state-of-the-art models are rapidly closing the gap with human-level performance on diverse computer vision and NLP tasks separately, they struggle to solve tasks that require joint reasoning over visual and textual modalities. Inspired by GLUE (Wang et. al., 2018)- a multitask benchmark for natural language understanding, we propose VL-GLUE in this paper. VL-GLUE consists of over 100k samples spanned across seven different tasks, which at their core require visuo-linguistic reasoning. Moreover, our benchmark comprises of diverse image types (from synthetically rendered figures, and day-to-day scenes to charts and complex diagrams) and includes a broad variety of domain-specific text (from cooking, politics, and sports to high-school curricula), demonstrating the need for multi-modal understanding in the real-world. We show that this benchmark is quite challenging for existing large-scale vision-language models and encourage development of systems that possess robust visuo-linguistic reasoning capabilities.