AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

TL;DR

AlignMMBench is a comprehensive benchmark designed to evaluate Chinese multimodal alignment in large vision-language models, focusing on nuanced, real-world scenarios and multi-turn dialogues to better assess model robustness and capabilities.

Contribution

This paper introduces AlignMMBench, the first benchmark specifically for Chinese visual contexts, with a new evaluation pipeline and a quantitative alignment score for robustness assessment.

Findings

VLMs show varied performance across tasks

Benchmark reveals limitations in current models' robustness

AlignMMBench provides detailed insights into Chinese multimodal alignment

Abstract

Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, which provides more nuanced evaluations of alignment capabilities and is the first benchmark specifically designed for Chinese visual contexts. This benchmark is meticulously curated from real-world scenarios and internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we develop CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Additionally, we measure the "alignment score", a quantitative metric designed to assess the robustness and stability of models across diverse prompts. Finally, we evaluate the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. The evaluation code and data are available at https://github.com/THUDM/AlignMMBench.