Open-World Human-Object Interaction Detection via Multi-modal Prompts

TL;DR

This paper introduces MP-HOI, a multi-modal prompt-based detector for open-world human-object interaction detection, leveraging large-scale datasets and contrastive learning to improve generalization, zero-shot performance, and handle ambiguous interactions.

Contribution

The paper presents MP-HOI, a novel multi-modal prompt-based approach that integrates visual prompts and large-scale datasets for improved open-world HOI detection.

Findings

Surpasses existing models by over 30 times in HOI vocabulary coverage.

Achieves state-of-the-art results across multiple benchmarks.

Exhibits strong zero-shot generalization capabilities.

Abstract

In this paper, we develop \textbf{MP-HOI}, a powerful Multi-modal Prompt-based HOI detector designed to leverage both textual descriptions for open-set generalization and visual exemplars for handling high ambiguity in descriptions, realizing HOI detection in the open world. Specifically, it integrates visual prompts into existing language-guided-only HOI detectors to handle situations where textual descriptions face difficulties in generalization and to address complex scenarios with high interaction ambiguity. To facilitate MP-HOI training, we build a large-scale HOI dataset named Magic-HOI, which gathers six existing datasets into a unified label space, forming over 186K images with 2.4K objects, 1.2K actions, and 20K HOI interactions. Furthermore, to tackle the long-tail issue within the Magic-HOI dataset, we introduce an automated pipeline for generating realistically annotated HOI images and present SynHOI, a high-quality synthetic HOI dataset containing 100K images. Leveraging these two datasets, MP-HOI optimizes the HOI task as a similarity learning process between multi-modal prompts and objects/interactions via a unified contrastive loss, to learn generalizable and transferable objects/interactions representations from large-scale data. MP-HOI could serve as a generalist HOI detector, surpassing the HOI vocabulary of existing expert models by more than 30 times. Concurrently, our results demonstrate that MP-HOI exhibits remarkable zero-shot capability in real-world scenarios and consistently achieves a new state-of-the-art performance across various benchmarks.