What and When to Look?: Temporal Span Proposal Network for Video Relation Detection

TL;DR

This paper introduces the Temporal Span Proposal Network (TSPN), a novel method for video relation detection that predicts relation categories and temporal spans, improving efficiency and performance over existing approaches.

Contribution

The paper proposes TSPN, which sparsifies relation search and predicts temporal spans simultaneously, addressing limitations of previous segment- and window-based methods in VidVRD.

Findings

TSPN accelerates training by over 2 times compared to existing methods.

TSPN achieves competitive results on VidVRD benchmarks.

Ablative experiments validate the effectiveness of TSPN.

Abstract

Identifying relations between objects is central to understanding the scene. While several works have been proposed for relation modeling in the image domain, there have been many constraints in the video domain due to challenging dynamics of spatio-temporal interactions (e.g., between which objects are there an interaction? when do relations start and end?). To date, two representative methods have been proposed to tackle Video Visual Relation Detection (VidVRD): segment-based and window-based. We first point out limitations of these methods and propose a novel approach named Temporal Span Proposal Network (TSPN). TSPN tells what to look: it sparsifies relation search space by scoring relationness of object pair, i.e., measuring how probable a relation exist. TSPN tells when to look: it simultaneously predicts start-end timestamps (i.e., temporal spans) and categories of the all possible relations by utilizing full video context. These two designs enable a win-win scenario: it accelerates training by 2X or more than existing methods and achieves competitive performance on two VidVRD benchmarks (ImageNet-VidVDR and VidOR). Moreover, comprehensive ablative experiments demonstrate the effectiveness of our approach. Codes are available at https://github.com/sangminwoo/Temporal-Span-Proposal-Network-VidVRD.