ViLBias: Detecting and Reasoning about Bias in Multimodal Content

TL;DR

ViLBias introduces a comprehensive benchmark and framework for detecting and reasoning about bias in multimodal news content, leveraging large language and vision-language models to improve accuracy and interpretability.

Contribution

The paper presents a new multimodal bias detection benchmark with a large annotated dataset and evaluates various models and tuning strategies for bias detection and reasoning tasks.

Findings

Incorporating images improves bias detection accuracy by 3-5%.

LLMs and VLMs outperform small models in capturing subtle biases.

Parameter-efficient tuning recovers 97-99% of full fine-tuning performance.

Abstract

Detecting bias in multimodal news requires models that reason over text--image pairs, not just classify text. In response, we present ViLBias, a VQA-style benchmark and framework for detecting and reasoning about bias in multimodal news. The dataset comprises 40,945 text--image pairs from diverse outlets, each annotated with a bias label and concise rationale using a two-stage LLM-as-annotator pipeline with hierarchical majority voting and human-in-the-loop validation. We evaluate Small Language Models (SLMs), Large Language Models (LLMs), and Vision--Language Models (VLMs) across closed-ended classification and open-ended reasoning (oVQA), and compare parameter-efficient tuning strategies. Results show that incorporating images alongside text improves detection accuracy by 3--5\%, and that LLMs/VLMs better capture subtle framing and text--image inconsistencies than SLMs. Parameter-efficient methods (LoRA/QLoRA/Adapters) recover 97--99\% of full fine-tuning performance with $<5\%$ trainable parameters. For oVQA, reasoning accuracy spans 52--79\% and faithfulness 68--89\%, both improved by instruction tuning; closed accuracy correlates strongly with reasoning ($r = 0.91$). ViLBias offers a scalable benchmark and strong baselines for multimodal bias detection and rationale quality.