Improving Panoptic Segmentation at All Scales

TL;DR

This paper introduces crop-aware loss and scale-aware data augmentation strategies to improve panoptic segmentation across multiple scales, achieving state-of-the-art results on several large datasets.

Contribution

The paper proposes a novel crop-aware bounding box regression loss and a scale-aware data sampling strategy to enhance panoptic segmentation performance on multi-megapixel images.

Findings

Achieved +4.5% PQ improvement on Mapillary Vistas dataset.

Surpassed previous best results on Indian Driving and Cityscapes datasets.

Demonstrated effectiveness of crop-aware training in handling large objects.

Abstract

Crop-based training strategies decouple training resolution from GPU memory consumption, allowing the use of large-capacity panoptic segmentation networks on multi-megapixel images. Using crops, however, can introduce a bias towards truncating or missing large objects. To address this, we propose a novel crop-aware bounding box regression loss (CABB loss), which promotes predictions to be consistent with the visible parts of the cropped objects, while not over-penalizing them for extending outside of the crop. We further introduce a novel data sampling and augmentation strategy which improves generalization across scales by counteracting the imbalanced distribution of object sizes. Combining these two contributions with a carefully designed, top-down panoptic segmentation architecture, we obtain new state-of-the-art results on the challenging Mapillary Vistas (MVD), Indian Driving and Cityscapes datasets, surpassing the previously best approach on MVD by +4.5% PQ and +5.2% mAP.