Hallucination Mitigation Prompts Long-term Video Understanding

TL;DR

This paper introduces a hallucination mitigation pipeline for long video understanding using multimodal large language models, improving accuracy and reducing hallucinations in long video question answering tasks.

Contribution

It presents a novel pipeline combining frame sampling, question-guided visual feature extraction, and answer generation techniques to mitigate hallucinations in long video understanding.

Findings

Achieved 84.2% accuracy on MovieChat dataset

Surpassed baseline by 29.1% in global mode

Won third place in CVPR LOVEU 2024 challenge

Abstract

Recently, multimodal large language models have made significant advancements in video understanding tasks. However, their ability to understand unprocessed long videos is very limited, primarily due to the difficulty in supporting the enormous memory overhead. Although existing methods achieve a balance between memory and information by aggregating frames, they inevitably introduce the severe hallucination issue. To address this issue, this paper constructs a comprehensive hallucination mitigation pipeline based on existing MLLMs. Specifically, we use the CLIP Score to guide the frame sampling process with questions, selecting key frames relevant to the question. Then, We inject question information into the queries of the image Q-former to obtain more important visual features. Finally, during the answer generation stage, we utilize chain-of-thought and in-context learning techniques to explicitly control the generation of answers. It is worth mentioning that for the breakpoint mode, we found that image understanding models achieved better results than video understanding models. Therefore, we aggregated the answers from both types of models using a comparison mechanism. Ultimately, We achieved 84.2\% and 62.9\% for the global and breakpoint modes respectively on the MovieChat dataset, surpassing the official baseline model by 29.1\% and 24.1\%. Moreover the proposed method won the third place in the CVPR LOVEU 2024 Long-Term Video Question Answering Challenge. The code is avaiable at https://github.com/lntzm/CVPR24Track-LongVideo