Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering

TL;DR

This paper introduces a temporal-aware activation engineering method to reduce hallucinations in VideoLLMs by focusing on temporal variation sensitivity, achieving significant improvements without extra fine-tuning.

Contribution

It is the first to explore activation engineering for hallucination mitigation in VideoLLMs, emphasizing temporal variation as a key factor and proposing a novel adaptive framework.

Findings

Significantly reduces hallucinations across multiple VideoLLMs and benchmarks.

Identifies temporal variation as a critical factor influencing hallucination sensitivity.

Demonstrates robustness of the proposed method without additional fine-tuning.

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding.However, hallucination, where the model generates plausible yet incorrect outputs, persists as a significant and under-addressed challenge in the video domain. Among existing solutions, activation engineering has proven successful in mitigating hallucinations in LLMs and ImageLLMs, yet its applicability to VideoLLMs remains largely unexplored. In this work, we are the first to systematically investigate the effectiveness and underlying mechanisms of activation engineering for mitigating hallucinations in VideoLLMs. We initially conduct an investigation of the key factors affecting the performance of activation engineering and find that a model's sensitivity to hallucination depends on $\textbf{temporal variation}$ rather than task type. Moreover, selecting appropriate internal modules and dataset for activation engineering is critical for reducing hallucination. Guided by these findings, we propose a temporal-aware activation engineering framework for VideoLLMs, which adaptively identifies and manipulates hallucination-sensitive modules based on the temporal variation characteristic, substantially mitigating hallucinations without additional LLM fine-tuning. Experiments across multiple models and benchmarks demonstrate that our method markedly reduces hallucination in VideoLLMs, thereby validating the robustness of our findings.