Cross-modal Contrastive Distillation for Instructional Activity Anticipation

TL;DR

This paper introduces a cross-modal contrastive distillation framework that leverages external textual knowledge to improve instructional activity anticipation from videos, generating natural language descriptions of future actions.

Contribution

It proposes a novel cross-modal contrastive distillation method to transfer knowledge between visual and textual modalities, enhancing anticipation accuracy.

Findings

40.2% relative improvement in BLEU4 score

Outperforms state-of-the-art methods on Tasty Videos dataset

Effectively utilizes external textual knowledge for visual prediction

Abstract

In this study, we aim to predict the plausible future action steps given an observation of the past and study the task of instructional activity anticipation. Unlike previous anticipation tasks that aim at action label prediction, our work targets at generating natural language outputs that provide interpretable and accurate descriptions of future action steps. It is a challenging task due to the lack of semantic information extracted from the instructional videos. To overcome this challenge, we propose a novel knowledge distillation framework to exploit the related external textual knowledge to assist the visual anticipation task. However, previous knowledge distillation techniques generally transfer information within the same modality. To bridge the gap between the visual and text modalities during the distillation process, we devise a novel cross-modal contrastive distillation (CCD) scheme, which facilitates knowledge distillation between teacher and student in heterogeneous modalities with the proposed cross-modal distillation loss. We evaluate our method on the Tasty Videos dataset. CCD improves the anticipation performance of the visual-alone student model by a large margin of 40.2% relatively in BLEU4. Our approach also outperforms the state-of-the-art approaches by a large margin.