Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

TL;DR

This paper evaluates the CLIP vision-language model for human action recognition across diverse domains, analyzing its limitations under masking strategies, and proposes a noise-based enhancement to improve its robustness and generalization in healthcare applications.

Contribution

It systematically assesses CLIP's performance on action recognition, identifies its limitations under masking, and introduces a novel noise-injection method to enhance its generalization capabilities.

Findings

CLIP shows inconsistent behavior with frequent misclassifications under masking.

Incorporating class-specific noise improves accuracy and reduces bias.

The approach offers potential for better generalization in healthcare scenarios.

Abstract

Human action recognition plays a critical role in healthcare and medicine, supporting applications such as patient behavior monitoring, fall detection, surgical robot supervision, and procedural skill assessment. While traditional models like CNNs and RNNs have achieved moderate success, they often struggle to generalize across diverse and complex actions. Recent advancements in vision-language models, especially the transformer-based CLIP model, offer promising capabilities for generalizing action recognition from video data. In this work, we evaluate CLIP on the UCF-101 dataset and systematically analyze its performance under three masking strategies: (1) percentage-based and shape-based black masking at 10%, 30%, and 50%, (2) feature-specific masking to suppress bias-inducing elements, and (3) isolation masking that retains only class-specific regions. Our results reveal that CLIP exhibits inconsistent behavior and frequent misclassifications, particularly when essential visual cues are obscured. To overcome these limitations, we propose incorporating class-specific noise, learned via a custom loss function, to reinforce attention to class-defining features. This enhancement improves classification accuracy and model confidence while reducing bias. We conclude with a discussion on the challenges of applying such models in clinical domains and outline directions for future work to improve generalizability across domain-independent healthcare scenarios.