On the Robustness of Human-Object Interaction Detection against Distribution Shift

TL;DR

This paper benchmarks the robustness of Human-Object Interaction detection models under distribution shifts, analyzes their weaknesses, and proposes simple, effective methods to enhance their robustness, improving practical applicability.

Contribution

It introduces the first robustness evaluation benchmark for HOI detection, analyzes existing models' weaknesses, and proposes plug-and-play strategies to improve robustness against distribution shifts.

Findings

Existing HOI models are insufficiently robust under distribution shifts.

The proposed data augmentation and feature fusion strategies significantly improve robustness.

The methods also enhance performance on standard benchmarks.

Abstract

Human-Object Interaction (HOI) detection has seen substantial advances in recent years. However, existing works focus on the standard setting with ideal images and natural distribution, far from practical scenarios with inevitable distribution shifts. This hampers the practical applicability of HOI detection. In this work, we investigate this issue by benchmarking, analyzing, and enhancing the robustness of HOI detection models under various distribution shifts. We start by proposing a novel automated approach to create the first robustness evaluation benchmark for HOI detection. Subsequently, we evaluate more than 40 existing HOI detection models on this benchmark, showing their insufficiency, analyzing the features of different frameworks, and discussing how the robustness in HOI is different from other tasks. With the insights from such analyses, we propose to improve the robustness of HOI detection methods through: (1) a cross-domain data augmentation integrated with mixup, and (2) a feature fusion strategy with frozen vision foundation models. Both are simple, plug-and-play, and applicable to various methods. Our experimental results demonstrate that the proposed approach significantly increases the robustness of various methods, with benefits on standard benchmarks, too. The dataset and code will be released.