Exploring Predicate Visual Context in Detecting Human-Object Interactions

TL;DR

This paper enhances human-object interaction detection by integrating predicate visual context through improved cross-attention mechanisms, leading to better performance on standard benchmarks without increasing training costs.

Contribution

It introduces a novel approach to incorporate predicate visual context into transformer-based HOI detectors using improved query design and spatial guidance, outperforming existing methods.

Findings

Outperforms state-of-the-art on HICO-DET and V-COCO benchmarks

Maintains low training cost despite improvements

Demonstrates the importance of visual context in complex interactions

Abstract

Recently, the DETR framework has emerged as the dominant approach for human--object interaction (HOI) research. In particular, two-stage transformer-based HOI detectors are amongst the most performant and training-efficient approaches. However, these often condition HOI classification on object features that lack fine-grained contextual information, eschewing pose and orientation information in favour of visual cues about object identity and box extremities. This naturally hinders the recognition of complex or ambiguous interactions. In this work, we study these issues through visualisations and carefully designed experiments. Accordingly, we investigate how best to re-introduce image features via cross-attention. With an improved query design, extensive exploration of keys and values, and box pair positional embeddings as spatial guidance, our model with enhanced predicate visual context (PViC) outperforms state-of-the-art methods on the HICO-DET and V-COCO benchmarks, while maintaining low training cost.