One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognition

TL;DR

This paper introduces CineMEC, a multimodal approach for entity coreference in videos that enhances understanding by linking textual descriptions with visual grounding, improving captioning and grounding accuracy.

Contribution

It proposes CineMEC, a multi-stage method that unites event role mentions with visual clusters without explicit grounding supervision, leveraging the synergy between grounding and captioning.

Findings

CineMEC improves captioning metrics (+2.5% CIDEr, +7% LEA)

CineMEC enhances grounding accuracy (+18% HOTA)

Extended VidSitu dataset with grounding annotations

Abstract

Video Situation Recognition (VidSitu) addresses the challenging problem of "who did what to whom, with what, how, and where" in a video. It tests thorough video understanding by requiring identification of salient actions and associated short descriptions for event roles across multiple events. Grounding with VidSitu requires spatio-temporal localization of key entities across shots and varied appearances. We posit that coherent video understanding requires consistent identification of entities that play different roles. We propose Multimodal Entity Coreference (MEC) to unite entity descriptions in text with grounding across the video. Towards this, we introduce CineMEC, a multi-stage approach that unites event role mention groups with visual clusters of entities, without explicit grounding supervision during training. Our approach is designed to exploit the synergy between visual grounding and captioning, where improving one influences the other and vice versa. For evaluation, we extend the VidSitu dataset with grounding annotations. While previous work focuses primarily on descriptions, CineMEC improves consistency across both: captioning (+2.5% CIDEr, +7% LEA) and visual grounding (+18% HOTA).