ODE: Open-Set Evaluation of Hallucinations in Multimodal Large Language Models

TL;DR

This paper introduces ODE, a dynamic open-set evaluation protocol for hallucinations in multimodal large language models, addressing static benchmark limitations and revealing higher hallucination rates with generated samples.

Contribution

We propose ODE, a novel graph-based, open-set evaluation protocol that enhances hallucination assessment in MLLMs and mitigates data contamination risks.

Findings

MLLMs show increased hallucination rates with ODE-generated samples.

ODE effectively reveals hallucination patterns and aids in model fine-tuning.

The protocol is applicable to both general and specialized scenarios.

Abstract

Hallucination poses a persistent challenge for multimodal large language models (MLLMs). However, existing benchmarks for evaluating hallucinations are generally static, which may overlook the potential risk of data contamination. To address this issue, we propose ODE, an open-set, dynamic protocol designed to evaluate object hallucinations in MLLMs at both the existence and attribute levels. ODE employs a graph-based structure to represent real-world object concepts, their attributes, and the distributional associations between them. This structure facilitates the extraction of concept combinations based on diverse distributional criteria, generating varied samples for structured queries that evaluate hallucinations in both generative and discriminative tasks. Through the generation of new samples, dynamic concept combinations, and varied distribution frequencies, ODE mitigates the risk of data contamination and broadens the scope of evaluation. This protocol is applicable to both general and specialized scenarios, including those with limited data. Experimental results demonstrate the effectiveness of our protocol, revealing that MLLMs exhibit higher hallucination rates when evaluated with ODE-generated samples, which indicates potential data contamination. Furthermore, these generated samples aid in analyzing hallucination patterns and fine-tuning models, offering an effective approach to mitigating hallucinations in MLLMs.