Can Hallucination Correction Improve Video-Language Alignment?

TL;DR

This paper proposes HACA, a self-training framework that corrects hallucinations in video descriptions to improve video-language alignment, leading to better performance in captioning and retrieval tasks.

Contribution

Introducing a novel hallucination correction-based training method that enhances video-language alignment in large vision-language models.

Findings

Improved video-caption binding accuracy

Enhanced text-to-video retrieval performance

Consistent gains across multiple evaluation metrics

Abstract

Large Vision-Language Models often generate hallucinated content that is not grounded in its visual inputs. While prior work focuses on mitigating hallucinations, we instead explore leveraging hallucination correction as a training objective to improve video-language alignment. We introduce HACA, a self-training framework learning to correct hallucinations in descriptions that do not align with the video content. By identifying and correcting inconsistencies, HACA enhances the model's ability to align video and textual representations for spatio-temporal reasoning. Our experimental results show consistent gains in video-caption binding and text-to-video retrieval tasks, demonstrating that hallucination correction-inspired tasks serve as an effective strategy for improving vision and language alignment.