Prototype Embedding Optimization for Human-Object Interaction Detection in Livestreaming

TL;DR

This paper introduces PeO-HOI, a novel prototype embedding optimization method that enhances human-object interaction detection in livestreaming by reducing object bias and improving accuracy.

Contribution

It proposes a new prototype embedding optimization approach specifically designed for HOI detection in livestreaming, addressing object bias issues present in existing methods.

Findings

Achieved 37.19% full detection accuracy on VidHOI dataset.

Improved HOI detection performance on self-built BJUT-HOI dataset.

Effectively mitigated object bias in livestreaming HOI detection.

Abstract

Livestreaming often involves interactions between streamers and objects, which is critical for understanding and regulating web content. While human-object interaction (HOI) detection has made some progress in general-purpose video downstream tasks, when applied to recognize the interaction behaviors between a streamer and different objects in livestreaming, it tends to focuses too much on the objects and neglects their interactions with the streamer, which leads to object bias. To solve this issue, we propose a prototype embedding optimization for human-object interaction detection (PeO-HOI). First, the livestreaming is preprocessed using object detection and tracking techniques to extract features of the human-object (HO) pairs. Then, prototype embedding optimization is adopted to mitigate the effect of object bias on HOI. Finally, after modelling the spatio-temporal context between HO pairs, the HOI detection results are obtained by the prediction head. The experimental results show that the detection accuracy of the proposed PeO-HOI method has detection accuracies of 37.19%@full, 51.42%@non-rare, 26.20%@rare on the publicly available dataset VidHOI, 45.13%@full, 62.78%@non-rare and 30.37%@rare on the self-built dataset BJUT-HOI, which effectively improves the HOI detection performance in livestreaming.