PCIE_Interaction Solution for Ego4D Social Interaction Challenge

TL;DR

This paper introduces the PCIE_Interaction solution for the Ego4D Social Interaction Challenge, combining face quality enhancement, ensemble methods, and audio-visual fusion to improve social interaction detection accuracy.

Contribution

It presents a novel approach that fuses visual and audio cues with face quality assessment for social interaction detection in egocentric videos.

Findings

Achieved 0.81 mAP on LAM task

Achieved 0.71 mAP on TTM task

Effective fusion of audio and visual cues

Abstract

This report presents our team's PCIE_Interaction solution for the Ego4D Social Interaction Challenge at CVPR 2025, addressing both Looking At Me (LAM) and Talking To Me (TTM) tasks. The challenge requires accurate detection of social interactions between subjects and the camera wearer, with LAM relying exclusively on face crop sequences and TTM combining speaker face crops with synchronized audio segments. In the LAM track, we employ face quality enhancement and ensemble methods. For the TTM task, we extend visual interaction analysis by fusing audio and visual cues, weighted by a visual quality score. Our approach achieved 0.81 and 0.71 mean average precision (mAP) on the LAM and TTM challenges leader board. Code is available at https://github.com/KanokphanL/PCIE_Ego4D_Social_Interaction