Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models

TL;DR

This paper presents a novel framework to reduce temporal hallucinations in multimodal large language models by extracting event-specific information and predicting event timestamps, thereby improving their understanding of video content.

Contribution

The study introduces a new method that decomposes event queries and employs models like CLIP and BLIP2 to enhance temporal understanding in MLLMs, significantly reducing hallucinations.

Findings

Reduced temporal hallucinations in MLLMs

Improved accuracy in event timestamp prediction

Enhanced response quality for temporal questions

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced the comprehension of multimedia content, bringing together diverse modalities such as text, images, and videos. However, a critical challenge faced by these models, especially when processing video inputs, is the occurrence of hallucinations - erroneous perceptions or interpretations, particularly at the event level. This study introduces an innovative method to address event-level hallucinations in MLLMs, focusing on specific temporal understanding in video content. Our approach leverages a novel framework that extracts and utilizes event-specific information from both the event query and the provided video to refine MLLMs' response. We propose a unique mechanism that decomposes on-demand event queries into iconic actions. Subsequently, we employ models like CLIP and BLIP2 to predict specific timestamps for event occurrences. Our evaluation, conducted using the Charades-STA dataset, demonstrates a significant reduction in temporal hallucinations and an improvement in the quality of event-related responses. This research not only provides a new perspective in addressing a critical limitation of MLLMs but also contributes a quantitatively measurable method for evaluating MLLMs in the context of temporal-related questions.