SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living

TL;DR

This paper introduces SKI models that integrate 3D skeleton data into vision-language embeddings, improving zero-shot activity recognition and captioning for daily living videos without needing skeleton data during inference.

Contribution

The paper presents SKI models that incorporate skeleton information into vision-language models via collaborative training, enabling better generalization to unseen activities.

Findings

Enhanced zero-shot activity recognition accuracy

Improved video captioning performance on ADL datasets

Skeleton integration boosts model robustness without skeleton data at inference

Abstract

The introduction of vision-language models like CLIP has enabled the development of foundational video models capable of generalizing to unseen videos and human actions. However, these models are typically trained on web videos, which often fail to capture the challenges present in Activities of Daily Living (ADL) videos. Existing works address ADL-specific challenges, such as similar appearances, subtle motion patterns, and multiple viewpoints, by combining 3D skeletons and RGB videos. However, these approaches are not integrated with language, limiting their ability to generalize to unseen action classes. In this paper, we introduce SKI models, which integrate 3D skeletons into the vision-language embedding space. SKI models leverage a skeleton-language model, SkeletonCLIP, to infuse skeleton information into Vision Language Models (VLMs) and Large Vision Language Models (LVLMs) through collaborative training. Notably, SKI models do not require skeleton data during inference, enhancing their robustness for real-world applications. The effectiveness of SKI models is validated on three popular ADL datasets for zero-shot action recognition and video caption generation tasks.