Hybrid-Learning Video Moment Retrieval across Multi-Domain Labels

TL;DR

This paper proposes a hybrid-learning approach for video moment retrieval that transfers knowledge from fully-supervised to weakly-supervised domains using a multi-branch model to improve retrieval accuracy without requiring extensive temporal annotations.

Contribution

It introduces EVA, a multi-branch video-text alignment model that enables knowledge transfer across domains with different label types for improved video moment retrieval.

Findings

EVA effectively leverages source domain annotations to enhance target domain retrieval.

The model achieves improved performance in weakly-labelled settings.

Cross-modal feature alignment enhances domain-invariant representations.

Abstract

Video moment retrieval (VMR) is to search for a visual temporal moment in an untrimmed raw video by a given text query description (sentence). Existing studies either start from collecting exhaustive frame-wise annotations on the temporal boundary of target moments (fully-supervised), or learn with only the video-level video-text pairing labels (weakly-supervised). The former is poor in generalisation to unknown concepts and/or novel scenes due to restricted dataset scale and diversity under expensive annotation costs; the latter is subject to visual-textual mis-correlations from incomplete labels. In this work, we introduce a new approach called hybrid-learning video moment retrieval to solve the problem by knowledge transfer through adapting the video-text matching relationships learned from a fully-supervised source domain to a weakly-labelled target domain when they do not share a common label space. Our aim is to explore shared universal knowledge between the two domains in order to improve model learning in the weakly-labelled target domain. Specifically, we introduce a multiplE branch Video-text Alignment model (EVA) that performs cross-modal (visual-textual) matching information sharing and multi-modal feature alignment to optimise domain-invariant visual and textual features as well as per-task discriminative joint video-text representations. Experiments show EVA's effectiveness in exploring temporal segment annotations in a source domain to help learn video moment retrieval without temporal labels in a target domain.