Video Moment Localization using Object Evidence and Reverse Captioning

TL;DR

This paper introduces MML, an advanced model for language-based video moment localization that incorporates object evidence and reverse captioning to improve accuracy over existing methods.

Contribution

The paper proposes MML, an extension of MAC, integrating object segmentation masks and video captioning features, along with enhanced language modeling, to better localize moments in videos based on complex language queries.

Findings

MML outperforms MAC baseline by 4.93% on R@1

MML outperforms MAC baseline by 1.70% on R@5

Demonstrates improved localization accuracy on Charades-STA dataset

Abstract

We address the problem of language-based temporal localization of moments in untrimmed videos. Compared to temporal localization with fixed categories, this problem is more challenging as the language-based queries have no predefined activity classes and may also contain complex descriptions. Current state-of-the-art model MAC addresses it by mining activity concepts from both video and language modalities. This method encodes the semantic activity concepts from the verb/object pair in a language query and leverages visual activity concepts from video activity classification prediction scores. We propose "Multi-faceted VideoMoment Localizer" (MML), an extension of MAC model by the introduction of visual object evidence via object segmentation masks and video understanding features via video captioning. Furthermore, we improve language modelling in sentence embedding. We experimented on Charades-STA dataset and identified that MML outperforms MAC baseline by 4.93% and 1.70% on R@1 and R@5metrics respectively. Our code and pre-trained model are publicly available at https://github.com/madhawav/MML.