Memory-Guided Semantic Learning Network for Temporal Sentence Grounding

TL;DR

This paper introduces MGSL-Net, a memory-augmented network for temporal sentence grounding that improves recognition of rare cases by memorizing and retrieving cross-modal semantic features, enhancing generalization and accuracy.

Contribution

The paper proposes a novel memory-guided network with a memory augmentation module to better handle rare cases in TSG tasks, improving model robustness and performance.

Findings

Significantly improves accuracy on rare cases in TSG benchmarks.

Enhances generalization by memorizing cross-modal semantic features.

Outperforms existing methods in effectiveness and efficiency.

Abstract

Temporal sentence grounding (TSG) is crucial and fundamental for video understanding. Although the existing methods train well-designed deep networks with a large amount of data, we find that they can easily forget the rarely appeared cases in the training stage due to the off-balance data distribution, which influences the model generalization and leads to undesirable performance. To tackle this issue, we propose a memory-augmented network, called Memory-Guided Semantic Learning Network (MGSL-Net), that learns and memorizes the rarely appeared content in TSG tasks. Specifically, MGSL-Net consists of three main parts: a cross-modal inter-action module, a memory augmentation module, and a heterogeneous attention module. We first align the given video-query pair by a cross-modal graph convolutional network, and then utilize a memory module to record the cross-modal shared semantic features in the domain-specific persistent memory. During training, the memory slots are dynamically associated with both common and rare cases, alleviating the forgetting issue. In testing, the rare cases can thus be enhanced by retrieving the stored memories, resulting in better generalization. At last, the heterogeneous attention module is utilized to integrate the enhanced multi-modal features in both video and query domains. Experimental results on three benchmarks show the superiority of our method on both effectiveness and efficiency, which substantially improves the accuracy not only on the entire dataset but also on rare cases.