G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o

TL;DR

G-VEval is a new evaluation metric for image and video captioning that leverages GPT-4o's reasoning capabilities, supporting multiple modes and outperforming existing metrics in correlating with human judgments.

Contribution

The paper introduces G-VEval, a versatile, multimodal evaluation metric powered by GPT-4o, and proposes MSVD-Eval, a new dataset for more transparent captioning assessment.

Findings

G-VEval outperforms existing metrics in correlation with human annotations.

Supports reference-free, reference-only, and combined evaluation modes.

Establishes a new dataset, MSVD-Eval, for comprehensive video captioning evaluation.

Abstract

Evaluation metric of visual captioning is important yet not thoroughly explored. Traditional metrics like BLEU, METEOR, CIDEr, and ROUGE often miss semantic depth, while trained metrics such as CLIP-Score, PAC-S, and Polos are limited in zero-shot scenarios. Advanced Language Model-based metrics also struggle with aligning to nuanced human preferences. To address these issues, we introduce G-VEval, a novel metric inspired by G-Eval and powered by the new GPT-4o. G-VEval uses chain-of-thought reasoning in large multimodal models and supports three modes: reference-free, reference-only, and combined, accommodating both video and image inputs. We also propose MSVD-Eval, a new dataset for video captioning evaluation, to establish a more transparent and consistent framework for both human experts and evaluation metrics. It is designed to address the lack of clear criteria in existing datasets by introducing distinct dimensions of Accuracy, Completeness, Conciseness, and Relevance (ACCR). Extensive results show that G-VEval outperforms existing methods in correlation with human annotations, as measured by Kendall tau-b and Kendall tau-c. This provides a flexible solution for diverse captioning tasks and suggests a straightforward yet effective approach for large language models to understand video content, paving the way for advancements in automated captioning. Codes are available at https://github.com/ztangaj/gveval