Multi-modal, Multi-task, Multi-criteria Automatic Evaluation with Vision Language Models

TL;DR

HarmonicEval is a new multi-criteria, reference-free evaluation metric for vision-language models that better aligns with human judgments across multiple tasks, supported by a large human-annotated benchmark.

Contribution

The paper introduces HarmonicEval, a novel multi-criteria evaluation metric, and the MMHE benchmark, enabling more accurate and adaptable assessment of vision-language models across diverse tasks.

Findings

HarmonicEval outperforms traditional metrics in correlating with human judgments.

The MMHE benchmark includes 18,000 expert annotations across four tasks.

HarmonicEval provides criterion-wise scores, enhancing interpretability.

Abstract

Vision-language models (VLMs) have shown impressive abilities across a range of multi-modal tasks. However, existing metrics for evaluating the quality of text generated by VLMs typically focus on an overall evaluation for a specific task, such as image captioning. While the overall evaluation is essential for any task, the criteria prioritized can differ depending on the task, making it challenging for current metrics to adapt to multi-task scenarios. To address this limitation, we propose HarmonicEval, a reference-free comprehensive evaluation metric that aggregates criterion-wise scores to produce the overall score in a bottom-up manner. Furthermore, to assess the generalizability of automatic evaluation metrics in multi-task scenarios, we construct the Multi-task Multi-criteria Human Evaluation (MMHE) benchmark, which comprises 18,000 expert human judgments across four multi-modal tasks. Our experiments demonstrate that HarmonicEval achieves higher correlations with human judgments than conventional metrics while providing numerical scores for each criterion. Project page: https://stjohn2007.github.io/MMHE_project/