FingerCap: Fine-grained Finger-level Hand Motion Captioning

TL;DR

This paper introduces FingerCap, a new dataset and method for generating detailed textual descriptions of finger-level hand motions, addressing the challenge of capturing subtle, high-frequency finger dynamics in videos.

Contribution

It presents FingerCap-40K, a large-scale dataset, and FiGOP, a novel temporal encoding method that improves fine-grained hand motion captioning.

Findings

FiGOP enhances motion understanding in Video-MLLMs.

Strong models still struggle with finger-level reasoning.

FiGOP yields consistent improvements in evaluations.

Abstract

Understanding fine-grained human hand motion is fundamental to visual perception, embodied intelligence, and multimodal communication. In this work, we propose Fine-grained Finger-level Hand Motion Captioning (FingerCap), which aims to generate textual descriptions that capture detailed finger-level semantics of hand actions. To support this task, we curate FingerCap-40K, a large-scale corpus of 40K paired hand-motion videos and captions spanning two complementary sources: concise instruction-style finger motions and diverse, naturalistic hand-object interactions. To enable effective evaluation, we employ HandJudge, a LLM-based rubric that measures finger-level correctness and motion completeness. Temporal sparsity remains a fundamental bottleneck for current Video-MLLMs, since sparse RGB sampling is insufficient to capture the subtle, high-frequency dynamics underlying fine finger motions. As a simple and compute-friendly remedy, we introduce FiGOP (Finger Group-of-Pictures), which pairs each RGB keyframe with subsequent hand keypoints until the next keyframe. A lightweight temporal encoder converts the keypoints into motion embeddings and integrates them with RGB features. FiGOP adapts the classic GOP concept to finger motion, recovering fine temporal cues without increasing RGB density. Experiments on FingerCap-40K show that strong open- and closed-source Video-MLLMs still struggle with finger-level reasoning, while our FiGOP-augmented model yield consistent gains under HandJudge and human studies.