OLAF: A Plug-and-Play Framework for Enhanced Multi-object Multi-part Scene Parsing

TL;DR

OLAF is a flexible framework that enhances multi-object multi-part scene segmentation by augmenting input data with structural cues and introducing a guidance encoder, leading to significant performance improvements across models and datasets.

Contribution

OLAF introduces a plug-and-play method with input augmentation and a new encoder module, improving segmentation accuracy for complex multi-object multi-part scenes.

Findings

Achieves up to 4.0 mIoU gain on Pascal-Parts-201

Improves performance across CNN, U-Net, and Transformer architectures

Demonstrates broad applicability and significant accuracy improvements

Abstract

Multi-object multi-part scene segmentation is a challenging task whose complexity scales exponentially with part granularity and number of scene objects. To address the task, we propose a plug-and-play approach termed OLAF. First, we augment the input (RGB) with channels containing object-based structural cues (fg/bg mask, boundary edge mask). We propose a weight adaptation technique which enables regular (RGB) pre-trained models to process the augmented (5-channel) input in a stable manner during optimization. In addition, we introduce an encoder module termed LDF to provide low-level dense feature guidance. This assists segmentation, particularly for smaller parts. OLAF enables significant mIoU gains of $\mathbf{3.3}$ (Pascal-Parts-58), $\mathbf{3.5}$ (Pascal-Parts-108) over the SOTA model. On the most challenging variant (Pascal-Parts-201), the gain is $\mathbf{4.0}$. Experimentally, we show that OLAF's broad applicability enables gains across multiple architectures (CNN, U-Net, Transformer) and datasets. The code is available at olafseg.github.io