Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration

TL;DR

This paper introduces INP-CC, an innovative end-to-end method for open-vocabulary human-object interaction detection that uses interaction-aware prompts and concept calibration to improve detection of novel interactions.

Contribution

The paper presents a novel interaction-aware prompt generator and language model-guided concept calibration for better HOI detection beyond training classes.

Findings

Outperforms state-of-the-art on SWIG-HOI and HICO-DET datasets.

Enhances differentiation of similar HOI concepts.

Improves detection of unseen interaction classes.

Abstract

Open Vocabulary Human-Object Interaction (HOI) detection aims to detect interactions between humans and objects while generalizing to novel interaction classes beyond the training set. Current methods often rely on Vision and Language Models (VLMs) but face challenges due to suboptimal image encoders, as image-level pre-training does not align well with the fine-grained region-level interaction detection required for HOI. Additionally, effectively encoding textual descriptions of visual appearances remains difficult, limiting the model's ability to capture detailed HOI relationships. To address these issues, we propose INteraction-aware Prompting with Concept Calibration (INP-CC), an end-to-end open-vocabulary HOI detector that integrates interaction-aware prompts and concept calibration. Specifically, we propose an interaction-aware prompt generator that dynamically generates a compact set of prompts based on the input scene, enabling selective sharing among similar interactions. This approach directs the model's attention to key interaction patterns rather than generic image-level semantics, enhancing HOI detection. Furthermore, we refine HOI concept representations through language model-guided calibration, which helps distinguish diverse HOI concepts by investigating visual similarities across categories. A negative sampling strategy is also employed to improve inter-modal similarity modeling, enabling the model to better differentiate visually similar but semantically distinct actions. Extensive experimental results demonstrate that INP-CC significantly outperforms state-of-the-art models on the SWIG-HOI and HICO-DET datasets. Code is available at https://github.com/ltttpku/INP-CC.