Advancing Compressed Video Action Recognition through Progressive Knowledge Distillation

TL;DR

This paper introduces Progressive Knowledge Distillation (PKD) and WISE, novel methods for improving compressed video action recognition by transferring knowledge across modalities and combining classifier outputs, leading to significant accuracy gains.

Contribution

The paper proposes PKD and WISE, innovative techniques for knowledge transfer and ensemble inference in compressed video action recognition, enhancing accuracy and generalization.

Findings

PKD improves IC accuracy by up to 11.42% on HMDB-51.

WISE boosts accuracy by up to 9.30% on HMDB-51.

Training ICs with PKD outperforms standard cross-entropy training.

Abstract

Compressed video action recognition classifies video samples by leveraging the different modalities in compressed videos, namely motion vectors, residuals, and intra-frames. For this purpose, three neural networks are deployed, each dedicated to processing one modality. Our observations indicate that the network processing intra-frames tend to converge to a flatter minimum than the network processing residuals, which in turn converges to a flatter minimum than the motion vector network. This hierarchy in convergence motivates our strategy for knowledge transfer among modalities to achieve flatter minima, which are generally associated with better generalization. With this insight, we propose Progressive Knowledge Distillation (PKD), a technique that incrementally transfers knowledge across the modalities. This method involves attaching early exits (Internal Classifiers - ICs) to the three networks. PKD distills knowledge starting from the motion vector network, followed by the residual, and finally, the intra-frame network, sequentially improving IC accuracy. Further, we propose the Weighted Inference with Scaled Ensemble (WISE), which combines outputs from the ICs using learned weights, boosting accuracy during inference. Our experiments demonstrate the effectiveness of training the ICs with PKD compared to standard cross-entropy-based training, showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and HMDB-51 datasets, respectively. Additionally, WISE improves accuracy by up to 4.28% and 9.30% on UCF-101 and HMDB-51, respectively.