VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living

TL;DR

VPN++ introduces a novel approach to enhance activity recognition by integrating pose knowledge into RGB features through distillation, achieving high accuracy and robustness without relying heavily on computationally expensive 3D pose estimation.

Contribution

The paper proposes VPN++, a unified model that combines feature-level and attention-level distillation to leverage pose information efficiently for ADL recognition.

Findings

VPN++ outperforms baseline methods on four datasets.

The approach is resilient to noisy pose data.

VPN++ offers high speed and accuracy without extensive 3D pose computation.

Abstract

Many attempts have been made towards combining RGB and 3D poses for the recognition of Activities of Daily Living (ADL). ADL may look very similar and often necessitate to model fine-grained details to distinguish them. Because the recent 3D ConvNets are too rigid to capture the subtle visual patterns across an action, this research direction is dominated by methods combining RGB and 3D Poses. But the cost of computing 3D poses from RGB stream is high in the absence of appropriate sensors. This limits the usage of aforementioned approaches in real-world applications requiring low latency. Then, how to best take advantage of 3D Poses for recognizing ADL? To this end, we propose an extension of a pose driven attention mechanism: Video-Pose Network (VPN), exploring two distinct directions. One is to transfer the Pose knowledge into RGB through a feature-level distillation and the other towards mimicking pose driven attention through an attention-level distillation. Finally, these two approaches are integrated into a single model, we call VPN++. We show that VPN++ is not only effective but also provides a high speed up and high resilience to noisy Poses. VPN++, with or without 3D Poses, outperforms the representative baselines on 4 public datasets. Code is available at https://github.com/srijandas07/vpnplusplus.