What I See Is What You See: Joint Attention Learning for First and Third Person Video Co-analysis

TL;DR

This paper introduces a novel joint attention learning method for analyzing and matching first- and third-person videos, leveraging a multi-branch deep network with triplet loss to improve cross-viewpoint understanding.

Contribution

It proposes a new joint attention framework with a multi-branch deep network and self-supervised learning for first- and third-person video co-analysis, outperforming existing methods.

Findings

Outperforms state-of-the-art in cross-viewpoint video matching

Effectively learns shared attention regions across viewpoints

Enhances applications through joint attention insights

Abstract

In recent years, more and more videos are captured from the first-person viewpoint by wearable cameras. Such first-person video provides additional information besides the traditional third-person video, and thus has a wide range of applications. However, techniques for analyzing the first-person video can be fundamentally different from those for the third-person video, and it is even more difficult to explore the shared information from both viewpoints. In this paper, we propose a novel method for first- and third-person video co-analysis. At the core of our method is the notion of "joint attention", indicating the learnable representation that corresponds to the shared attention regions in different viewpoints and thus links the two viewpoints. To this end, we develop a multi-branch deep network with a triplet loss to extract the joint attention from the first- and third-person videos via self-supervised learning. We evaluate our method on the public dataset with cross-viewpoint video matching tasks. Our method outperforms the state-of-the-art both qualitatively and quantitatively. We also demonstrate how the learned joint attention can benefit various applications through a set of additional experiments.