GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval

TL;DR

GMMFormer introduces an efficient, implicit clip modeling approach using Gaussian-Mixture-Models within a Transformer architecture for partially relevant video retrieval, improving semantic discrimination and reducing redundancy.

Contribution

It proposes GMMFormer, a novel Transformer-based model that models video clips implicitly with Gaussian-Mixture-Models and enhances semantic differentiation with a query diverse loss.

Findings

Outperforms existing methods on three large-scale datasets.

Reduces storage overhead compared to scanning-based clip construction.

Achieves higher retrieval accuracy and efficiency.

Abstract

Given a text query, partially relevant video retrieval (PRVR) seeks to find untrimmed videos containing pertinent moments in a database. For PRVR, clip modeling is essential to capture the partial relationship between texts and videos. Current PRVR methods adopt scanning-based clip construction to achieve explicit clip modeling, which is information-redundant and requires a large storage overhead. To solve the efficiency problem of PRVR methods, this paper proposes GMMFormer, a Gaussian-Mixture-Model based Transformer which models clip representations implicitly. During frame interactions, we incorporate Gaussian-Mixture-Model constraints to focus each frame on its adjacent frames instead of the whole video. Then generated representations will contain multi-scale clip information, achieving implicit clip modeling. In addition, PRVR methods ignore semantic differences between text queries relevant to the same video, leading to a sparse embedding space. We propose a query diverse loss to distinguish these text queries, making the embedding space more intensive and contain more semantic information. Extensive experiments on three large-scale video datasets (i.e., TVR, ActivityNet Captions, and Charades-STA) demonstrate the superiority and efficiency of GMMFormer. Code is available at \url{https://github.com/huangmozhi9527/GMMFormer}.