HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

TL;DR

HumaniBench is a comprehensive evaluation framework that assesses large multimodal models on human-centered principles like fairness, ethics, and empathy across diverse real-world visual tasks, revealing persistent gaps and trade-offs.

Contribution

This work introduces HumaniBench, a unified, large-scale evaluation framework for systematically measuring human-centric alignment in multimodal models.

Findings

Proprietary models excel in ethics, reasoning, and empathy.

Open-source models perform better in visual grounding and resilience.

All models show gaps in fairness and multilingual inclusivity.

Abstract

Although recent large multimodal models (LMMs) demonstrate impressive progress on vision language tasks, their alignment with human centered (HC) principles, such as fairness, ethics, inclusivity, empathy, and robustness; remains poorly understood. We present HumaniBench, a unified evaluation framework designed to characterize HC alignment across realistic, socially grounded visual contexts. HumaniBench contains 32,000 expert-verified image question pairs derived from real world news imagery and spanning seven evaluation tasks: scene understanding, instance identity, multiple-choice visual question answering (VQA), multilinguality, visual grounding, empathetic captioning, and image resilience testing. Each task is mapped to one or more HC principles through a principled operationalization of metrics covering accuracy, harmful content detection, hallucination and faithfulness, coherence, cross lingual quality, empathy, and robustness.We evaluate 15 state-of-the-art LMMs under this framework and observe consistent cross model trade offs: proprietary systems achieve the strongest performance on ethics, reasoning, and empathy, while open-source models exhibit superior visual grounding and resilience. All models, however, show persistent gaps in fairness and multilingual inclusivity. We further analyze the effect of inference-time techniques, finding that chain of thought prompting and test-time scaling yield 8 to 12 % improvements on several HC dimensions. HumaniBench provides a reproducible, extensible foundation for systematic HC evaluation of LMMs and enables fine-grained analysis of alignment trade-offs that are not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/