HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild
Supreeth Narasimhaswamy, Huy Anh Nguyen, Lihan Huang, Minh Hoai

TL;DR
HOIST-Former is a transformer-based model designed to accurately identify, segment, and track hand-held objects in challenging scenarios involving occlusion and rapid motion, supported by a new in-the-wild dataset.
Contribution
The paper introduces HOIST-Former, a novel architecture for hand-held object segmentation and tracking, and provides the first large-scale in-the-wild dataset for this task.
Findings
HOIST-Former outperforms existing methods on multiple datasets.
The contact loss improves segmentation accuracy.
The HOIST dataset enables robust evaluation of hand-held object tracking.
Abstract
We address the challenging task of identifying, segmenting, and tracking hand-held objects, which is crucial for applications such as human action segmentation and performance evaluation. This task is particularly challenging due to heavy occlusion, rapid motion, and the transitory nature of objects being hand-held, where an object may be held, released, and subsequently picked up again. To tackle these challenges, we have developed a novel transformer-based architecture called HOIST-Former. HOIST-Former is adept at spatially and temporally segmenting hands and objects by iteratively pooling features from each other, ensuring that the processes of identification, segmentation, and tracking of hand-held objects depend on the hands' positions and their contextual appearance. We further refine HOIST-Former with a contact loss that focuses on areas where hands are in contact with objects.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Pose and Action Recognition · Hand Gesture Recognition Systems · Video Surveillance and Tracking Methods
