Joint Forecasting of Panoptic Segmentations with Difference Attention

TL;DR

This paper introduces a transformer-based model with difference attention for joint panoptic segmentation forecasting, explicitly modeling object velocities and accelerations to improve accuracy and achieve state-of-the-art results.

Contribution

The paper proposes a novel joint forecasting model using difference attention in transformers, addressing independent object treatment and heuristic merging issues in prior work.

Findings

Achieves state-of-the-art panoptic segmentation forecasting metrics.

Difference attention effectively models object velocities and accelerations.

Refines predictions by incorporating depth estimates.

Abstract

Forecasting of a representation is important for safe and effective autonomy. For this, panoptic segmentations have been studied as a compelling representation in recent work. However, recent state-of-the-art on panoptic segmentation forecasting suffers from two issues: first, individual object instances are treated independently of each other; second, individual object instance forecasts are merged in a heuristic manner. To address both issues, we study a new panoptic segmentation forecasting model that jointly forecasts all object instances in a scene using a transformer model based on 'difference attention.' It further refines the predictions by taking depth estimates into account. We evaluate the proposed model on the Cityscapes and AIODrive datasets. We find difference attention to be particularly suitable for forecasting because the difference of quantities like locations enables a model to explicitly reason about velocities and acceleration. Because of this, we attain state-of-the-art on panoptic segmentation forecasting metrics.