A Real-Time System for Egocentric Hand-Object Interaction Detection in Industrial Domains

TL;DR

This paper introduces a real-time egocentric hand-object interaction detection system for industrial settings, combining action recognition and object detection modules to improve speed and accuracy.

Contribution

It presents a cascaded architecture with a novel action recognition model and a fine-tuned object detector, achieving high performance at 30fps in real-time industrial scenarios.

Findings

Mamba model achieves 38.52% p-AP on ENIGMA-51

YOLOWorld reaches 85.13% AP for hand and object detection

System operates effectively in real-time at 30fps

Abstract

Hand-object interaction detection remains an open challenge in real-time applications, where intuitive user experiences depend on fast and accurate detection of interactions with surrounding objects. We propose an efficient approach for detecting hand-objects interactions from streaming egocentric vision that operates in real time. Our approach consists of an action recognition module and an object detection module for identifying active objects upon confirmed interaction. Our Mamba model with EfficientNetV2 as backbone for action recognition achieves 38.52% p-AP on the ENIGMA-51 benchmark at 30fps, while our fine-tuned YOLOWorld reaches 85.13% AP for hand and object. We implement our models in a cascaded architecture where the action recognition and object detection modules operate sequentially. When the action recognition predicts a contact state, it activates the object detection module, which in turn performs inference on the relevant frame to detect and classify the active object.