DQEN: Dual Query Enhancement Network for DETR-based HOI Detection

TL;DR

DQEN introduces a dual query enhancement approach for DETR-based HOI detection, improving interaction understanding by enhancing object and interaction queries with semantic fusion and auxiliary prediction, leading to better performance on standard datasets.

Contribution

The paper proposes a novel Dual Query Enhancement Network (DQEN) that enhances object and interaction queries using object-aware features and semantic fusion, addressing limitations of random query initialization in DETR-based HOI detection.

Findings

Achieves competitive results on HICO-Det and V-COCO datasets.

Enhances interaction query initialization with CLIP-based semantic features.

Improves detection accuracy by auxiliary interaction feature prediction.

Abstract

Human-Object Interaction (HOI) detection focuses on localizing human-object pairs and recognizing their interactions. Recently, the DETR-based framework has been widely adopted in HOI detection. In DETR-based HOI models, queries with clear meaning are crucial for accurately detecting HOIs. However, prior works have typically relied on randomly initialized queries, leading to vague representations that limit the model's effectiveness. Meanwhile, humans in the HOI categories are fixed, while objects and their interactions are variable. Therefore, we propose a Dual Query Enhancement Network (DQEN) to enhance object and interaction queries. Specifically, object queries are enhanced with object-aware encoder features, enabling the model to focus more effectively on humans interacting with objects in an object-aware way. On the other hand, we design a novel Interaction Semantic Fusion module to exploit the HOI candidates that are promoted by the CLIP model. Semantic features are extracted to enhance the initialization of interaction queries, thereby improving the model's ability to understand interactions. Furthermore, we introduce an Auxiliary Prediction Unit aimed at improving the representation of interaction features. Our proposed method achieves competitive performance on both the HICO-Det and the V-COCO datasets. The source code is available at https://github.com/lzzhhh1019/DQEN.