DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioning

TL;DR

DENEB is a new automatic evaluation metric for image captioning that is specifically designed to be robust against hallucinations, utilizing a novel transformer-based approach and trained on a large, diverse dataset.

Contribution

We introduce DENEB, a supervised evaluation metric that effectively handles hallucinations in image captioning by processing multiple references simultaneously with the Sim-Vec Transformer.

Findings

DENEB outperforms existing metrics on multiple datasets.

DENEB demonstrates robustness against hallucinations.

The Nebula dataset supports effective training of DENEB.

Abstract

In this work, we address the challenge of developing automatic evaluation metrics for image captioning, with a particular focus on robustness against hallucinations. Existing metrics are often inadequate for handling hallucinations, primarily due to their limited ability to compare candidate captions with multifaceted reference captions. To address this shortcoming, we propose DENEB, a novel supervised automatic evaluation metric specifically robust against hallucinations. DENEB incorporates the Sim-Vec Transformer, a mechanism that processes multiple references simultaneously, thereby efficiently capturing the similarity between an image, a candidate caption, and reference captions. To train DENEB, we construct the diverse and balanced Nebula dataset comprising 32,978 images, paired with human judgments provided by 805 annotators. We demonstrated that DENEB achieves state-of-the-art performance among existing LLM-free metrics on the FOIL, Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and PASCAL-50S datasets, validating its effectiveness and robustness against hallucinations.