EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understanding

TL;DR

EgoIllusion is a new benchmark designed to evaluate hallucination tendencies in multimodal large language models when understanding egocentric videos, highlighting current challenges and guiding future improvements.

Contribution

This paper introduces EgoIllusion, the first benchmark specifically targeting hallucinations in egocentric video understanding by MLLMs, with comprehensive evaluation data.

Findings

Current MLLMs achieve only 59% accuracy on hallucination detection.

Significant hallucination challenges remain in egocentric video understanding.

Benchmark will be open-sourced for community use.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in complex multimodal tasks. While MLLMs excel at visual perception and reasoning in third-person and egocentric videos, they are prone to hallucinations, generating coherent yet inaccurate responses. We present EgoIllusion, a first benchmark to evaluate MLLM hallucinations in egocentric videos. EgoIllusion comprises 1,400 videos paired with 8,000 human-annotated open and closed-ended questions designed to trigger hallucinations in both visual and auditory cues in egocentric videos. Evaluations across ten MLLMs reveal significant challenges, including powerful models like GPT-4o and Gemini, achieving only 59% accuracy. EgoIllusion lays the foundation in developing robust benchmarks to evaluate the effectiveness of MLLMs and spurs the development of better egocentric MLLMs with reduced hallucination rates. Our benchmark will be open-sourced for reproducibility.