SDGBiasBench: Benchmarking and Mitigating Vision--Language Models' Biases in Sustainable Development Goals

TL;DR

SDGBiasBench is a comprehensive benchmark suite designed to evaluate and mitigate biases in vision-language models related to Sustainable Development Goals, highlighting intrinsic SDG biases and proposing a novel debiasing method.

Contribution

The paper introduces SDGBiasBench, a large-scale benchmark for SDG reasoning, and proposes CADE, a training-free debiasing method that improves model fairness and accuracy.

Findings

Current VLMs show SDG-specific bias driven by priors rather than evidence.

CADE improves multiple-choice accuracy by up to 25%.

CADE reduces regression MAE by up to 12 points.

Abstract

Assessing progress toward the Sustainable Development Goals (SDGs) requires multi-step reasoning over visual cues, contextual knowledge, and development indicators, where incomplete evidence use and imperfect evidence integration can introduce hidden prediction biases. Real-world SDG monitoring further spans both qualitative judgments and quantitative estimation. However, existing benchmarks typically evaluate these aspects in isolation, obscuring systematic biases that emerge when models substitute priors for evidence. To address this gap, we propose SDGBiasBench, a large-scale benchmark suite for SDG-oriented vision-language reasoning. Spanning 500k expert-involved multiple-choice questions and 50k regression tasks, the benchmark enables comprehensive assessment of both decision-level and estimation-level bias in Vision--Language Models (VLMs). Evaluations on SDGBiasBench reveal an intrinsic SDG bias in current VLMs, where predictions are frequently driven by SDG specific priors rather than reliable multi-modal cues. To mitigate such bias, we propose CADE (Contrastive Adaptive Debias Ensemble), a training-free, plug-and-play method that leverages modality-specific answer priors. CADE yields significant gains on the proposed benchmark, improving multiple-choice accuracy by up to 25% and reducing regression MAE by up to 12 points across multiple VLMs. We hope our work can foster the development of more fair and reliable AI systems for sustainable development.