CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification

TL;DR

This paper introduces CPFD, a novel confidence-aware feature distillation method that enhances short video classification by adaptively transferring privileged dense features, improving performance and stability over existing methods.

Contribution

The paper proposes CPFD, a confidence-based distillation approach that adaptively weights privileged features during training, outperforming traditional uniform-weight methods and reducing performance gaps.

Findings

CPFD improves video classification F1 score by 6.76% over end-to-end models.

CPFD reduces the performance gap between teacher and student models by 84.6%.

The method is effective in diverse tasks and has been deployed in production systems.

Abstract

Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-intensive and less accessible during online inference. Consequently, these dense features are categorized as `Privileged Dense Features'.Meanwhile, end-to-end multi-modal models have shown promising results in numerous computer vision tasks. In industrial applications, prioritizing end-to-end multi-modal features, can enhance efficiency but often leads to the loss of valuable information from historical privileged dense features. To integrate both features while maintaining efficiency and manageable resource costs, we present Confidence-aware Privileged Feature Distillation (CPFD), which empowers features of an end-to-end multi-modal model by adaptively distilling privileged features during training. Unlike existing privileged feature distillation (PFD) methods, which apply uniform weights to all instances during distillation, potentially causing unstable performance across different business scenarios and a notable performance gap between teacher model (Dense Feature enhanced multimodal-model DF-X-VLM) and student model (multimodal-model only X-VLM), our CPFD leverages confidence scores derived from the teacher model to adaptively mitigate the performance variance with the student model. We conducted extensive offline experiments on five diverse tasks demonstrating that CPFD improves the video classification F1 score by 6.76% compared with end-to-end multimodal-model (X-VLM) and by 2.31% with vanilla PFD on-average. And it reduces the performance gap by 84.6% and achieves results comparable to teacher model DF-X-VLM. The effectiveness of CPFD is further substantiated by online experiments, and our framework has been deployed in production systems for over a dozen models.