Preserve Pre-trained Knowledge: Transfer Learning With Self-Distillation For Action Recognition

TL;DR

This paper introduces a self-distillation based transfer learning method for action recognition that effectively preserves pre-trained knowledge during fine-tuning, outperforming existing methods on standard datasets.

Contribution

It proposes a novel self-distillation approach that fixes the encoder from the last epoch as a teacher to retain pre-trained knowledge during transfer learning.

Findings

Outperforms state-of-the-art on UCF101 and HMDB51 datasets.

Effectively mitigates catastrophic forgetting in transfer learning.

Simple yet effective training strategy.

Abstract

Video-based action recognition is one of the most popular topics in computer vision. With recent advances of selfsupervised video representation learning approaches, action recognition usually follows a two-stage training framework, i.e., self-supervised pre-training on large-scale unlabeled sets and transfer learning on a downstream labeled set. However, catastrophic forgetting of the pre-trained knowledge becomes the main issue in the downstream transfer learning of action recognition, resulting in a sub-optimal solution. In this paper, to alleviate the above issue, we propose a novel transfer learning approach that combines self-distillation in fine-tuning to preserve knowledge from the pre-trained model learned from the large-scale dataset. Specifically, we fix the encoder from the last epoch as the teacher model to guide the training of the encoder from the current epoch in the transfer learning. With such a simple yet effective learning strategy, we outperform state-of-the-art methods on widely used UCF101 and HMDB51 datasets in action recognition task.