Learning Universal Shape Dictionary for Realtime Instance Segmentation

TL;DR

This paper introduces USD-Seg, an explicit shape representation method for instance segmentation using a learned shape dictionary, achieving real-time performance and high accuracy on COCO dataset.

Contribution

It proposes a universal shape dictionary for explicit shape modeling in instance segmentation, enabling fast and explainable shape predictions with minimal overhead.

Findings

Achieves 35.8 AP at 65 fps with YOLOv4 on COCO.

Attains 34.1 AP at 12 fps with FCOS on COCO.

Introduces the Efficiency of AP (AP_E) metric for computational assessment.

Abstract

We present a novel explicit shape representation for instance segmentation. Based on how to model the object shape, current instance segmentation systems can be divided into two categories, implicit and explicit models. The implicit methods, which represent the object mask/contour by intractable network parameters, and produce it through pixel-wise classification, are predominant. However, the explicit methods, which parameterize the shape with simple and explainable models, are less explored. Since the operations to generate the final shape are light-weighted, the explicit methods have a clear speed advantage over implicit methods, which is crucial for real-world applications. The proposed USD-Seg adopts a linear model, sparse coding with dictionary, for object shapes. First, it learns a dictionary from a large collection of shape datasets, making any shape being able to be decomposed into a linear combination through the dictionary. Hence the name "Universal Shape Dictionary". Then it adds a simple shape vector regression head to ordinary object detector, giving the detector segmentation ability with minimal overhead. For quantitative evaluation, we use both average precision (AP) and the proposed Efficiency of AP (AP$_E$) metric, which intends to also measure the computational consumption of the framework to cater to the requirements of real-world applications. We report experimental results on the challenging COCO dataset, in which our single model on a single Titan Xp GPU achieves 35.8 AP and 27.8 AP$_E$ at 65 fps with YOLOv4 as base detector, 34.1 AP and 28.6 AP$_E$ at 12 fps with FCOS as base detector.