Exploring State Change Capture of Heterogeneous Backbones @ Ego4D Hands and Objects Challenge 2022

TL;DR

This paper presents a method using heterogeneous backbones for classifying object state changes and localizing their temporal boundaries in videos, achieving top performance in the Ego4D challenge.

Contribution

It introduces a novel approach combining CSN and VideoMAE backbones for improved accuracy in human-object interaction understanding.

Findings

Achieved 0.796 accuracy on OSCC

Achieved 0.516 temporal localization error on PNR

Ranked 1st on Ego4D leaderboard

Abstract

Capturing the state changes of interacting objects is a key technology for understanding human-object interactions. This technical report describes our method using heterogeneous backbones for the Ego4D Object State Change Classification and PNR Temporal Localization Challenge. In the challenge, we used the heterogeneous video understanding backbones, namely CSN with 3D convolution as operator and VideoMAE with Transformer as operator. Our method achieves an accuracy of 0.796 on OSCC while achieving an absolute temporal localization error of 0.516 on PNR. These excellent results rank 1st on the leaderboard of Ego4D OSCC & PNR-TL Challenge 2022.