Panoptic Scene Graph Generation

TL;DR

This paper introduces panoptic scene graph generation (PSG), a new approach that uses panoptic segmentation instead of bounding boxes to create more comprehensive scene graphs, along with a new dataset and baseline models.

Contribution

The paper proposes PSG as a novel task, creates a high-quality dataset, and develops baseline models including Transformer-based methods for improved scene understanding.

Findings

PSG dataset contains 49k annotated images from COCO and Visual Genome.

Two Transformer-based models, PSGTR and PSGFormer, outperform traditional baselines.

PSG enables more detailed and context-aware scene graph generation.

Abstract

Existing research addresses scene graph generation (SGG) -- a critical technology for scene understanding in images -- from a detection perspective, i.e., objects are detected using bounding boxes followed by prediction of their pairwise relationships. We argue that such a paradigm causes several problems that impede the progress of the field. For instance, bounding box-based labels in current datasets usually contain redundant classes like hairs, and leave out background information that is crucial to the understanding of context. In this work, we introduce panoptic scene graph generation (PSG), a new problem task that requires the model to generate a more comprehensive scene graph representation based on panoptic segmentations rather than rigid bounding boxes. A high-quality PSG dataset, which contains 49k well-annotated overlapping images from COCO and Visual Genome, is created for the community to keep track of its progress. For benchmarking, we build four two-stage baselines, which are modified from classic methods in SGG, and two one-stage baselines called PSGTR and PSGFormer, which are based on the efficient Transformer-based detector, i.e., DETR. While PSGTR uses a set of queries to directly learn triplets, PSGFormer separately models the objects and relations in the form of queries from two Transformer decoders, followed by a prompting-like relation-object matching mechanism. In the end, we share insights on open challenges and future directions.