Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion

TL;DR

This paper introduces a novel dense semantic forecasting method in video that predicts future pixel-level semantics by jointly regressing features and feature motion, achieving state-of-the-art accuracy across multiple dense prediction tasks.

Contribution

The paper presents a new joint regression approach combining feature and motion prediction, applicable to various architectures and tasks, with a decoupled, task-agnostic design.

Findings

Achieves state-of-the-art accuracy in semantic forecasting

Effective across semantic, instance, and panoptic segmentation tasks

Utilizes deformable convolutions and spatial correlation for improved predictions

Abstract

Dense semantic forecasting anticipates future events in video by inferring pixel-level semantics of an unobserved future image. We present a novel approach that is applicable to various single-frame architectures and tasks. Our approach consists of two modules. Feature-to-motion (F2M) module forecasts a dense deformation field that warps past features into their future positions. Feature-to-feature (F2F) module regresses the future features directly and is therefore able to account for emergent scenery. The compound F2MF model decouples the effects of motion from the effects of novelty in a task-agnostic manner. We aim to apply F2MF forecasting to the most subsampled and the most abstract representation of a desired single-frame model. Our design takes advantage of deformable convolutions and spatial correlation coefficients across neighbouring time instants. We perform experiments on three dense prediction tasks: semantic segmentation, instance-level segmentation, and panoptic segmentation. The results reveal state-of-the-art forecasting accuracy across three dense prediction tasks.