EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding
Yue Xu, Yong-Lu Li, Zhemin Huang, Michael Xu Liu, Cewu Lu, Yu-Wing, Tai, Chi-Keung Tang

TL;DR
EgoPCA introduces a comprehensive framework for egocentric hand-object interaction recognition, addressing domain gaps and setting new benchmarks with curated datasets and effective training strategies.
Contribution
The paper proposes a novel framework combining probing, curation, and adaptation to improve Ego-HOI recognition, including new datasets, baseline models, and training strategies.
Findings
Achieved state-of-the-art performance on Ego-HOI benchmarks.
Developed comprehensive pre-train and test datasets.
Established effective mechanisms for future research.
Abstract
With the surge in attention to Egocentric Hand-Object Interaction (Ego-HOI), large-scale datasets such as Ego4D and EPIC-KITCHENS have been proposed. However, most current research is built on resources derived from third-person video action recognition. This inherent domain gap between first- and third-person action videos, which have not been adequately addressed before, makes current Ego-HOI suboptimal. This paper rethinks and proposes a new framework as an infrastructure to advance Ego-HOI recognition by Probing, Curation and Adaption (EgoPCA). We contribute comprehensive pre-train sets, balanced test sets and a new baseline, which are complete with a training-finetuning strategy. With our new framework, we not only achieve state-of-the-art performance on Ego-HOI benchmarks but also build several new and effective mechanisms and settings to advance further research. We believe our…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understanding· youtube
Taxonomy
TopicsHuman Pose and Action Recognition · Stroke Rehabilitation and Recovery · Action Observation and Synchronization
