Contrastive learning-based video quality assessment-jointed video vision transformer for video recognition

TL;DR

This paper introduces SSL-V3, a novel self-supervised learning framework combining Video Quality Assessment with video classification using a Video Vision Transformer, improving accuracy especially in healthcare video datasets.

Contribution

The paper proposes a joint learning approach that integrates VQA into video classification, addressing label scarcity and enhancing robustness of video recognition models.

Findings

Achieved 94.87% accuracy on I-CONECT dataset

Effectively links VQA scores with classification features

Demonstrates robustness across two datasets

Abstract

Video quality significantly affects video classification. We found this problem when we classified Mild Cognitive Impairment well from clear videos, but worse from blurred ones. From then, we realized that referring to Video Quality Assessment (VQA) may improve video classification. This paper proposed Self-Supervised Learning-based Video Vision Transformer combined with No-reference VQA for video classification (SSL-V3) to fulfill the goal. SSL-V3 leverages Combined-SSL mechanism to join VQA into video classification and address the label shortage of VQA, which commonly occurs in video datasets, making it impossible to provide an accurate Video Quality Score. In brief, Combined-SSL takes video quality score as a factor to directly tune the feature map of the video classification. Then, the score, as an intersected point, links VQA and classification, using the supervised classification task to tune the parameters of VQA. SSL-V3 achieved robust experimental results on two datasets. For example, it reached an accuracy of 94.87% on some interview videos in the I-CONECT (a facial video-involved healthcare dataset), verifying SSL-V3's effectiveness.