HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamics

TL;DR

HanDyVQA is a new fine-grained video question-answering benchmark that captures detailed hand-object interaction dynamics, challenging current models and highlighting areas for improvement in spatial, motion, and part-level understanding.

Contribution

Introduces HanDyVQA, a comprehensive benchmark with diverse question types, segmentation masks, and analysis of model performance on fine-grained HOI reasoning.

Findings

Current models reach only 73% accuracy, far below human performance.

Explicit HOI cues improve model accuracy.

Challenges remain in spatial, motion, and part-level reasoning.

Abstract

Hand-object interaction (HOI) inherently involves dynamics where human manipulations produce distinct spatio-temporal effects on objects. However, existing semantic HOI benchmarks focused either on manipulation or on the resulting effects at a coarse level, lacking fine-grained spatio-temporal reasoning to capture the underlying dynamics in HOI. We introduce HanDyVQA, a fine-grained video question-answering benchmark that comprehensively covers both the manipulation and effect aspects of HOI. HanDyVQA comprises six complementary question types (Action, Process, Objects, Location, State Change, and Object Parts), totalling 11.1K multiple-choice QA pairs. Collected QA pairs recognizing manipulation styles, hand/object motions, and part-level state changes. HanDyVQA also includes 10.3K segmentation masks for Objects and Object Parts questions, enabling the evaluation of object/part-level reasoning in video object segmentation. We evaluated recent video foundation models on our benchmark and found that even the best-performing model, Gemini-2.5-Pro, reached only 73% average accuracy, which is far from human performance (97%). Further analysis shows the remaining challenges in spatial relationship, motion, and part-level geometric understanding. We also found that integrating explicit HOI-related cues into visual features improves performance, offering insights for developing future models with a deeper understanding of HOI dynamics.