Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection

TL;DR

This paper introduces a novel cross-modal knowledge distillation framework that enhances RGB representations for action detection by transferring temporal and contextual knowledge from additional modalities, improving performance with only RGB at inference.

Contribution

The paper proposes a two-level distillation method that transfers both atomic and sequence-level temporal knowledge from multiple modalities to RGB for action detection.

Findings

Outperforms existing cross-modal distillation methods in action detection.

Achieves competitive results with only RGB during inference.

Demonstrates the effectiveness of multi-level knowledge transfer for temporal understanding.

Abstract

In video understanding, most cross-modal knowledge distillation (KD) methods are tailored for classification tasks, focusing on the discriminative representation of the trimmed videos. However, action detection requires not only categorizing actions, but also localizing them in untrimmed videos. Therefore, transferring knowledge pertaining to temporal relations is critical for this task which is missing in the previous cross-modal KD frameworks. To this end, we aim at learning an augmented RGB representation for action detection, taking advantage of additional modalities at training time through KD. We propose a KD framework consisting of two levels of distillation. On one hand, atomic-level distillation encourages the RGB student to learn the sub-representation of the actions from the teacher in a contrastive manner. On the other hand, sequence-level distillation encourages the student to learn the temporal knowledge from the teacher, which consists of transferring the Global Contextual Relations and the Action Boundary Saliency. The result is an Augmented-RGB stream that can achieve competitive performance as the two-stream network while using only RGB at inference time. Extensive experimental analysis shows that our proposed distillation framework is generic and outperforms other popular cross-modal distillation methods in action detection task.