An Evaluation of Large Pre-Trained Models for Gesture Recognition using Synthetic Videos

TL;DR

This paper evaluates the effectiveness of synthetic videos and large pre-trained models for gesture recognition, finding that synthetic data alone underperforms real data, and zero-shot text classification is ineffective for gestures.

Contribution

It demonstrates the limitations of synthetic data and zero-shot methods for gesture recognition using large pre-trained video models, highlighting the importance of fine-tuned features.

Findings

Synthetic videos yield lower accuracy than real videos.

Fine-tuned video backbones outperform pre-trained ones.

Zero-shot text classification performs poorly on gestures.

Abstract

In this work, we explore the possibility of using synthetically generated data for video-based gesture recognition with large pre-trained models. We consider whether these models have sufficiently robust and expressive representation spaces to enable "training-free" classification. Specifically, we utilize various state-of-the-art video encoders to extract features for use in k-nearest neighbors classification, where the training data points are derived from synthetic videos only. We compare these results with another training-free approach -- zero-shot classification using text descriptions of each gesture. In our experiments with the RoCoG-v2 dataset, we find that using synthetic training videos yields significantly lower classification accuracy on real test videos compared to using a relatively small number of real training videos. We also observe that video backbones that were fine-tuned on classification tasks serve as superior feature extractors, and that the choice of fine-tuning data has a substantial impact on k-nearest neighbors performance. Lastly, we find that zero-shot text-based classification performs poorly on the gesture recognition task, as gestures are not easily described through natural language.