Distilling Knowledge from CNN-Transformer Models for Enhanced Human Action Recognition

TL;DR

This paper explores enhancing human action recognition by distilling knowledge from CNN and Transformer models, demonstrating significant accuracy improvements through a novel combination of local and global feature extraction techniques.

Contribution

It introduces a knowledge distillation framework using Transformer-based student models and CNN teacher models, improving performance in human action recognition tasks.

Findings

Knowledge distillation improves accuracy and mAP in action recognition.

Transformer-based models effectively capture global image dependencies.

Combining CNN and Transformer features enhances recognition performance.

Abstract

This paper presents a study on improving human action recognition through the utilization of knowledge distillation, and the combination of CNN and ViT models. The research aims to enhance the performance and efficiency of smaller student models by transferring knowledge from larger teacher models. The proposed method employs a Transformer vision network as the student model, while a convolutional network serves as the teacher model. The teacher model extracts local image features, whereas the student model focuses on global features using an attention mechanism. The Vision Transformer (ViT) architecture is introduced as a robust framework for capturing global dependencies in images. Additionally, advanced variants of ViT, namely PVT, Convit, MVIT, Swin Transformer, and Twins, are discussed, highlighting their contributions to computer vision tasks. The ConvNeXt model is introduced as a teacher model, known for its efficiency and effectiveness in computer vision. The paper presents performance results for human action recognition on the Stanford 40 dataset, comparing the accuracy and mAP of student models trained with and without knowledge distillation. The findings illustrate that the suggested approach significantly improves the accuracy and mAP when compared to training networks under regular settings. These findings emphasize the potential of combining local and global features in action recognition tasks.