SemCo: Toward Semantic Coherent Visual Relationship Forecasting

TL;DR

This paper introduces SemCoBench, a benchmark emphasizing semantic coherence in visual relationship forecasting, and proposes SemCoFormer, a transformer-based model with modules to improve understanding of object interactions in videos.

Contribution

It presents a new benchmark for semantic coherence in VRF and a novel transformer-based model with modules to better distinguish relationships and focus on their dynamics.

Findings

Model achieves improved accuracy on SemCoBench.

Semantic coherence modeling enhances relationship prediction.

Modules effectively distinguish similar relationships.

Abstract

Visual Relationship Forecasting (VRF) aims to anticipate relations among objects without observing future visual content. The task relies on capturing and modeling the semantic coherence in object interactions, as it underpins the evolution of events and scenes in videos. However, existing VRF datasets offer limited support for learning such coherence due to noisy annotations in the datasets and weak correlations between different actions and relationship transitions in subject-object pair. Furthermore, existing methods struggle to distinguish similar relationships and overfit to unchanging relationships in consecutive frames. To address these challenges, we present SemCoBench, a benchmark that emphasizes semantic coherence for visual relationship forecasting. Based on action labels and short-term subject-object pairs, SemCoBench decomposes relationship categories and dynamics by cleaning and reorganizing video datasets to ensure predicting semantic coherence in object interactions. In addition, we also present Semantic Coherent Transformer method (SemCoFormer) to model the semantic coherence with a Relationship Augmented Module (RAM) and a Coherence Reasoning Module (CRM). RAM is designed to distinguish similar relationships, and CRM facilitates the model's focus on the dynamics in relationships. The experimental results on SemCoBench demonstrate that modeling the semantic coherence is a key step toward reasonable, fine-grained, and diverse visual relationship forecasting, contributing to a more comprehensive understanding of video scenes.