Probing Visual Language Priors in VLMs

TL;DR

This paper introduces ViLP, a benchmark to evaluate visual reasoning in VLMs using out-of-distribution images and questions, revealing their over-reliance on language priors and proposing a self-training method to improve visual reasoning capabilities.

Contribution

The paper presents ViLP, a novel benchmark for testing visual reasoning in VLMs, and a self-improving training framework that enhances models' focus on visual inputs.

Findings

GPT-4 scores 66.17% on ViLP, indicating room for improvement.

Self-training with generated data boosts VLM performance.

Models trained with our method outperform baseline models on ViLP.

Abstract

Despite recent advances in Vision-Language Models (VLMs), they may over-rely on visual language priors existing in their training data rather than true visual reasoning. To investigate this, we introduce ViLP, a benchmark featuring deliberately out-of-distribution images synthesized via image generation models and out-of-distribution Q&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although, humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT-4 achieves only 66.17% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training. Our training objectives compel VLMs to focus more on the actual visual inputs, and we demonstrate their effectiveness in boosting the performance of open-source VLMs, including LLaVA-v1.5 and Cambrian.