Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition

TL;DR

This paper introduces a decoupled, zero-shot HOI detection framework using MLLMs that formulates interaction recognition as a deterministic visual question answering task, enabling flexible, training-free generalization to unseen interactions.

Contribution

It proposes a detector-agnostic, zero-shot HOI detection method leveraging MLLMs, with a deterministic IR formulation and a spatial-aware pooling module for improved performance.

Findings

Achieves superior zero-shot performance on HICO-DET and V-COCO datasets.

Demonstrates strong cross-dataset generalization.

Allows integration with any object detector without retraining.

Abstract

Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions. While advances in open-vocabulary object detection provide promising solutions for object localization, interaction recognition (IR) remains challenging due to the combinatorial diversity of interactions. Existing methods, including two-stage methods, tightly couple IR with a specific detector and rely on coarse-grained vision-language model (VLM) features, which limit generalization to unseen interactions. In this work, we propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR. We introduce a deterministic generation method that formulates IR as a visual question answering task and enforces deterministic outputs, enabling training-free zero-shot IR. To further enhance performance and efficiency by fine-tuning the model, we design a spatial-aware pooling module that integrates appearance and pairwise spatial cues, and a one-pass deterministic matching method that predicts all candidate interactions in a single forward pass. Extensive experiments on HICO-DET and V-COCO demonstrate that our method achieves superior zero-shot performance, strong cross-dataset generalization, and the flexibility to integrate with any object detectors without retraining. The codes are publicly available at https://github.com/SY-Xuan/DA-HOI.