Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts

TL;DR

This paper introduces a generator-evaluator neural network framework for panoptic image segmentation that can handle unfamiliar objects and parts, demonstrating competitive results on the COCO benchmark.

Contribution

It presents a novel generator-evaluator approach with class-agnostic segmentation capable of splitting unseen objects into parts.

Findings

Achieved results comparable to Mask-RCNN on COCO Panoptic benchmark.

Enabled splitting of unseen object classes into parts.

Demonstrated the effectiveness of generator-evaluator in segmentation tasks.

Abstract

In machine learning and other fields, suggesting a good solution to a problem is usually a harder task than evaluating the quality of such a solution. This asymmetry is the basis for a large number of selection oriented methods that use a generator system to guess a set of solutions and an evaluator system to rank and select the best solutions. This work examines the use of this approach to the problem of panoptic image segmentation and class agnostic parts segmentation. The generator/evaluator approach for this case consists of two independent convolutional neural nets: a generator net that suggests variety segments corresponding to objects, stuff and parts regions in the image, and an evaluator net that chooses the best segments to be merged into the segmentation map. The result is a trial and error evolutionary approach in which a generator that guesses segments with low average accuracy, but with wide variability, can still produce good results when coupled with an accurate evaluator. The generator consists of a Pointer net that receives an image and a point in the image, and predicts the region of the segment containing the point. Generating and evaluating each segment separately is essential in this case since it demands exponentially fewer guesses compared to a system that guesses and evaluates the full segmentation map in each try. The classification of the selected segments is done by an independent region-specific classification net. This allows the segmentation to be class agnostic and hence, capable of segmenting unfamiliar categories that were not part of the training set. The method was examined on the COCO Panoptic segmentation benchmark and gave results comparable to those of the basic semantic segmentation and Mask-RCNN methods. In addition, the system was used for the task of splitting objects of unseen classes (that did not appear in the training set) into parts.