Consistency-Aware Graph Network for Human Interaction Understanding

TL;DR

This paper introduces a consistency-aware graph network that enhances human interaction understanding by modeling complex relations and enforcing consistency, leading to improved performance on benchmark datasets.

Contribution

It proposes a novel consistency-aware graph network with a specialized reasoning module for better modeling of human interactions.

Findings

Achieves state-of-the-art results on three benchmarks.

Effectively models third-order interactive relations.

Demonstrates the benefit of consistency-aware reasoning.

Abstract

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main cause is that recent approaches learn human interactive relations via shallow graphical models, which is inadequate to model complicated human interactions. In this paper, we propose a consistency-aware graph network, which combines the representative ability of graph network and the consistency-aware reasoning to facilitate the HIU task. Our network consists of three components, a backbone CNN to extract image features, a factor graph network to learn third-order interactive relations among participants, and a consistency-aware reasoning module to enforce labeling and grouping consistencies. Our key observation is that the consistency-aware-reasoning bias for HIU can be embedded into an energy function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained jointly in an end-to-end manner. Experimental results show that our approach achieves leading performance on three benchmarks.