LGDN: Language-Guided Denoising Network for Video-Language Modeling

TL;DR

LGDN is a novel video-language model that dynamically filters noisy frames guided by language, focusing on salient content to improve cross-modal alignment and outperform existing methods on multiple datasets.

Contribution

The paper introduces LGDN, a new model that effectively filters irrelevant video frames using language guidance, addressing noise issues in video-language modeling.

Findings

LGDN outperforms state-of-the-art methods on five datasets.

Filtering noisy frames improves cross-modal alignment.

Ablation studies highlight the importance of noise reduction.

Abstract

Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically correlated, and focus on video-language modeling at video level. However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single video-level description; (2) A raw video typically has noisy/meaningless information (e.g., scenery shot, transition or teaser). Although a number of recent works deploy attention mechanism to alleviate this problem, the irrelevant/noisy information still makes it very difficult to address. To overcome such challenge, we thus propose an efficient and effective model, termed Language-Guided Denoising Network (LGDN), for video-language modeling. Different from most existing methods that utilize all extracted video frames, LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2--4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins. We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-language work.