Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation

TL;DR

Ada-Segment introduces an automated multi-loss adaptation method for panoptic segmentation, dynamically adjusting training losses to improve performance without manual tuning, achieving state-of-the-art results on COCO and ADE20K datasets.

Contribution

It proposes a novel automated multi-loss adaptation framework that models learning dynamics and generalizes across datasets for panoptic segmentation.

Findings

Achieves 2.7% PQ improvement on COCO val split

Sets new state-of-the-art 48.5% PQ on COCO test-dev

Attains 32.9% PQ on ADE20K

Abstract

Panoptic segmentation that unifies instance segmentation and semantic segmentation has recently attracted increasing attention. While most existing methods focus on designing novel architectures, we steer toward a different perspective: performing automated multi-loss adaptation (named Ada-Segment) on the fly to flexibly adjust multiple training losses over the course of training using a controller trained to capture the learning dynamics. This offers a few advantages: it bypasses manual tuning of the sensitive loss combination, a decisive factor for panoptic segmentation; it allows to explicitly model the learning dynamics, and reconcile the learning of multiple objectives (up to ten in our experiments); with an end-to-end architecture, it generalizes to different datasets without the need of re-tuning hyperparameters or re-adjusting the training process laboriously. Our Ada-Segment brings 2.7% panoptic quality (PQ) improvement on COCO val split from the vanilla baseline, achieving the state-of-the-art 48.5% PQ on COCO test-dev split and 32.9% PQ on ADE20K dataset. The extensive ablation studies reveal the ever-changing dynamics throughout the training process, necessitating the incorporation of an automated and adaptive learning strategy as presented in this paper.