FLOAT: Factorized Learning of Object Attributes for Improved Multi-object Multi-part Scene Parsing

TL;DR

FLOAT introduces a scalable factorized label space framework for multi-object multi-part scene parsing, significantly improving segmentation accuracy and handling diverse datasets with a novel inference-time zoom refinement technique.

Contribution

The paper proposes FLOAT, a novel factorized label space approach with a zoom refinement method, enhancing multi-object multi-part scene parsing performance and scalability.

Findings

Improves mIOU by 2-2.1% on Pascal-Part datasets.

Achieves 4.8-3.9% higher sqIOU compared to state-of-the-art.

Demonstrates effectiveness on a newly created comprehensive dataset Pascal-Part-201.

Abstract

Multi-object multi-part scene parsing is a challenging task which requires detecting multiple object classes in a scene and segmenting the semantic parts within each object. In this paper, we propose FLOAT, a factorized label space framework for scalable multi-object multi-part parsing. Our framework involves independent dense prediction of object category and part attributes which increases scalability and reduces task complexity compared to the monolithic label space counterpart. In addition, we propose an inference-time 'zoom' refinement technique which significantly improves segmentation quality, especially for smaller objects/parts. Compared to state of the art, FLOAT obtains an absolute improvement of 2.0% for mean IOU (mIOU) and 4.8% for segmentation quality IOU (sqIOU) on the Pascal-Part-58 dataset. For the larger Pascal-Part-108 dataset, the improvements are 2.1% for mIOU and 3.9% for sqIOU. We incorporate previously excluded part attributes and other minor parts of the Pascal-Part dataset to create the most comprehensive and challenging version which we dub Pascal-Part-201. FLOAT obtains improvements of 8.6% for mIOU and 7.5% for sqIOU on the new dataset, demonstrating its parsing effectiveness across a challenging diversity of objects and parts. The code and datasets are available at floatseg.github.io.