Exploiting Multimodal Synthetic Data for Egocentric Human-Object Interaction Detection in an Industrial Scenario

TL;DR

This paper introduces a synthetic multimodal dataset and a novel detection method for egocentric human-object interactions in industrial scenarios, demonstrating improved real-world performance through synthetic data pre-training.

Contribution

The paper presents EgoISM-HOI, a new synthetic dataset and a multimodal detection pipeline that enhances EHOI detection accuracy in industrial environments.

Findings

Synthetic data pre-training improves real-world EHOI detection performance.

The proposed method outperforms state-of-the-art class-agnostic approaches.

Public release of datasets and tools supports further research.

Abstract

In this paper, we tackle the problem of Egocentric Human-Object Interaction (EHOI) detection in an industrial setting. To overcome the lack of public datasets in this context, we propose a pipeline and a tool for generating synthetic images of EHOIs paired with several annotations and data signals (e.g., depth maps or segmentation masks). Using the proposed pipeline, we present EgoISM-HOI a new multimodal dataset composed of synthetic EHOI images in an industrial environment with rich annotations of hands and objects. To demonstrate the utility and effectiveness of synthetic EHOI data produced by the proposed tool, we designed a new method that predicts and combines different multimodal signals to detect EHOIs in RGB images. Our study shows that exploiting synthetic data to pre-train the proposed method significantly improves performance when tested on real-world data. Moreover, to fully understand the usefulness of our method, we conducted an in-depth analysis in which we compared and highlighted the superiority of the proposed approach over different state-of-the-art class-agnostic methods. To support research in this field, we publicly release the datasets, source code, and pre-trained models at https://iplab.dmi.unict.it/egoism-hoi.