Identity-Aware Multi-Sentence Video Description

TL;DR

This paper introduces a new task for multi-sentence video description that maintains consistent person identities across clips, utilizing a Transformer-based approach and gender-aware representations to improve re-identification and description coherence.

Contribution

It proposes the Fill-in the Identity auxiliary task and a two-stage approach for identity-aware video description, enhancing link consistency of persons across video segments.

Findings

The Fill-in the Identity model outperforms baselines and recent methods.

The approach enables coherent multi-sentence descriptions with re-identified persons.

Augmented LSMDC benchmark supports the new task and evaluation.

Abstract

Standard video and movie description tasks abstract away from person identities, thus failing to link identities across sentences. We propose a multi-sentence Identity-Aware Video Description task, which overcomes this limitation and requires to re-identify persons locally within a set of consecutive clips. We introduce an auxiliary task of Fill-in the Identity, that aims to predict persons' IDs consistently within a set of clips, when the video descriptions are given. Our proposed approach to this task leverages a Transformer architecture allowing for coherent joint prediction of multiple IDs. One of the key components is a gender-aware textual representation as well an additional gender prediction objective in the main model. This auxiliary task allows us to propose a two-stage approach to Identity-Aware Video Description. We first generate multi-sentence video descriptions, and then apply our Fill-in the Identity model to establish links between the predicted person entities. To be able to tackle both tasks, we augment the Large Scale Movie Description Challenge (LSMDC) benchmark with new annotations suited for our problem statement. Experiments show that our proposed Fill-in the Identity model is superior to several baselines and recent works, and allows us to generate descriptions with locally re-identified people.