VidLBEval: Benchmarking and Mitigating Language Bias in Video-Involved LVLMs

TL;DR

This paper introduces a benchmark and a mitigation method for addressing language bias in video-involved large vision-language models, revealing current models' limitations and proposing a solution that improves their fairness without retraining.

Contribution

It presents a new Video Language Bias Evaluation Benchmark and a Multi-branch Contrastive Decoding method to mitigate language bias in LVLMs without retraining.

Findings

Existing LVLMs are significantly biased towards language.

The proposed MCD method effectively reduces language bias.

MCD maintains model performance across various tasks.

Abstract

Recently, Large Vision-Language Models (LVLMs) have made significant strides across diverse multimodal tasks and benchmarks. This paper reveals a largely under-explored problem from existing video-involved LVLMs - language bias, where models tend to prioritize language over video and thus result in incorrect responses. To address this research gap, we first collect a Video Language Bias Evaluation Benchmark, which is specifically designed to assess the language bias in video-involved LVLMs through two key tasks: ambiguous video contrast and interrogative question probing. Accordingly, we design accompanied evaluation metrics that aim to penalize LVLMs being biased by language. In addition, we also propose Multi-branch Contrastive Decoding (MCD), introducing two expert branches to simultaneously counteract language bias potentially generated by the amateur text-only branch. Our experiments demonstrate that i) existing video-involved LVLMs, including both proprietary and open-sourced, are largely limited by the language bias problem; ii) our MCD can effectively mitigate this issue and maintain general-purpose capabilities in various video-involved LVLMs without any additional retraining or alteration to model architectures.