JEDI: Joint Expert Distillation in a Semi-Supervised Multi-Dataset Student-Teacher Scenario for Video Action Recognition

TL;DR

JEDI introduces a semi-supervised multi-dataset learning framework that leverages expert models to improve video action recognition across datasets through joint student-teacher training.

Contribution

It presents a novel joint expert distillation method that enhances generalization across datasets in a semi-supervised setting for video recognition.

Findings

Significant performance improvements over initial experts.

Effective multi-dataset generalization in video action recognition.

Joint end-to-end training enhances model robustness.

Abstract

We propose JEDI, a multi-dataset semi-supervised learning method, which efficiently combines knowledge from multiple experts, learned on different datasets, to train and improve the performance of individual, per dataset, student models. Our approach achieves this by addressing two important problems in current machine learning research: generalization across datasets and limitations of supervised training due to scarcity of labeled data. We start with an arbitrary number of experts, pretrained on their own specific dataset, which form the initial set of student models. The teachers are immediately derived by concatenating the feature representations from the penultimate layers of the students. We then train all models in a student-teacher semi-supervised learning scenario until convergence. In our efficient approach, student-teacher training is carried out jointly and end-to-end, showing that both students and teachers improve their generalization capacity during training. We validate our approach on four video action recognition datasets. By simultaneously considering all datasets within a unified semi-supervised setting, we demonstrate significant improvements over the initial experts.