DISCODE: Distribution-Aware Score Decoder for Robust Automatic Evaluation of Image Captioning

TL;DR

DISCODE is a novel, test-time adaptive evaluation method for image captioning that improves robustness and alignment with human judgments across diverse domains without requiring finetuning.

Contribution

It introduces DISCODE, a finetuning-free, distribution-aware evaluation approach using ATT loss and Gaussian prior, and presents MCEval, a new multi-domain caption evaluation benchmark.

Findings

DISCODE outperforms existing metrics on MCEval and other benchmarks.

It achieves state-of-the-art correlation with human judgments.

The method is robust under domain-shift scenarios.

Abstract

Large vision-language models (LVLMs) have shown impressive performance across a broad range of multimodal tasks. However, robust image caption evaluation using LVLMs remains challenging, particularly under domain-shift scenarios. To address this issue, we introduce the Distribution-Aware Score Decoder (DISCODE), a novel finetuning-free method that generates robust evaluation scores better aligned with human judgments across diverse domains. The core idea behind DISCODE lies in its test-time adaptive evaluation approach, which introduces the Adaptive Test-Time (ATT) loss, leveraging a Gaussian prior distribution to improve robustness in evaluation score estimation. This loss is efficiently minimized at test time using an analytical solution that we derive. Furthermore, we introduce the Multi-domain Caption Evaluation (MCEval) benchmark, a new image captioning evaluation benchmark covering six distinct domains, designed to assess the robustness of evaluation metrics. In our experiments, we demonstrate that DISCODE achieves state-of-the-art performance as a reference-free evaluation metric across MCEval and four representative existing benchmarks.