What-Meets-Where: Unified Learning of Action and Contact Localization in Images

TL;DR

This paper introduces PaIR-Net, a novel framework for jointly predicting action semantics and contact regions in images, supported by a new dataset, PaIR, to advance understanding of human-environment interactions.

Contribution

The paper proposes a unified model and dataset for simultaneous action and contact localization, addressing limitations of existing methods in capturing dual action semantics and spatial context.

Findings

PaIR-Net outperforms baseline models in contact and action prediction tasks.

The PaIR dataset contains 13,979 images with diverse actions, objects, and body parts.

Ablation studies validate the effectiveness of each architectural component.

Abstract

People control their bodies to establish contact with the environment. To comprehensively understand actions across diverse visual contexts, it is essential to simultaneously consider \textbf{what} action is occurring and \textbf{where} it is happening. Current methodologies, however, often inadequately capture this duality, typically failing to jointly model both action semantics and their spatial contextualization within scenes. To bridge this gap, we introduce a novel vision task that simultaneously predicts high-level action semantics and fine-grained body-part contact regions. Our proposed framework, PaIR-Net, comprises three key components: the Contact Prior Aware Module (CPAM) for identifying contact-relevant body parts, the Prior-Guided Concat Segmenter (PGCS) for pixel-wise contact segmentation, and the Interaction Inference Module (IIM) responsible for integrating global interaction relationships. To facilitate this task, we present PaIR (Part-aware Interaction Representation), a comprehensive dataset containing 13,979 images that encompass 654 actions, 80 object categories, and 17 body parts. Experimental evaluation demonstrates that PaIR-Net significantly outperforms baseline approaches, while ablation studies confirm the efficacy of each architectural component. The code and dataset will be released upon publication.