Compositional Learning in Transformer-Based Human-Object Interaction Detection

TL;DR

This paper introduces a transformer-based compositional learning framework for human-object interaction detection, improving generalization and performance on rare classes without auxiliary data.

Contribution

It presents a novel transformer-based approach for compositional HOI learning, surpassing CNN-based methods and enhancing rare class detection.

Findings

Achieves state-of-the-art performance on HOI detection benchmarks.

Improves detection of rare HOI classes.

Demonstrates the effectiveness of transformer-based compositional learning.

Abstract

Human-object interaction (HOI) detection is an important part of understanding human activities and visual scenes. The long-tailed distribution of labeled instances is a primary challenge in HOI detection, promoting research in few-shot and zero-shot learning. Inspired by the combinatorial nature of HOI triplets, some existing approaches adopt the idea of compositional learning, in which object and action features are learned individually and re-composed as new training samples. However, these methods follow the CNN-based two-stage paradigm with limited feature extraction ability, and often rely on auxiliary information for better performance. Without introducing any additional information, we creatively propose a transformer-based framework for compositional HOI learning. Human-object pair representations and interaction representations are re-composed across different HOI instances, which involves richer contextual information and promotes the generalization of knowledge. Experiments show our simple but effective method achieves state-of-the-art performance, especially on rare HOI classes.