GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models

TL;DR

This paper introduces GEM, a simple yet effective glass surface segmentation method leveraging vision foundation models, which constructs a large synthetic dataset and achieves state-of-the-art results without extensive manual labeling.

Contribution

It proposes a novel approach combining Stable Diffusion and SAM to create a synthetic dataset and a segmentation model, reducing manual effort and improving performance.

Findings

GEM surpasses previous methods with a 2.1% IoU improvement.

The synthetic dataset S-GSD performs well in zero-shot and transfer learning.

The approach reduces data annotation costs significantly.

Abstract

Detecting glass regions is a challenging task due to the inherent ambiguity in their transparency and reflective characteristics. Current solutions in this field remain rooted in conventional deep learning paradigms, requiring the construction of annotated datasets and the design of network architectures. However, the evident drawback with these mainstream solutions lies in the time-consuming and labor-intensive process of curating datasets, alongside the increasing complexity of model structures. In this paper, we propose to address these issues by fully harnessing the capabilities of two existing vision foundation models (VFMs): Stable Diffusion and Segment Anything Model (SAM). Firstly, we construct a Synthetic but photorealistic large-scale Glass Surface Detection dataset, dubbed S-GSD, without any labour cost via Stable Diffusion. This dataset consists of four different scales, consisting of 168k images totally with precise masks. Besides, based on the powerful segmentation ability of SAM, we devise a simple Glass surface sEgMentor named GEM, which follows the simple query-based encoder-decoder architecture. Comprehensive experiments are conducted on the large-scale glass segmentation dataset GSD-S. Our GEM establishes a new state-of-the-art performance with the help of these two VFMs, surpassing the best-reported method GlassSemNet with an IoU improvement of 2.1%. Additionally, extensive experiments demonstrate that our synthetic dataset S-GSD exhibits remarkable performance in zero-shot and transfer learning settings. Codes, datasets and models are publicly available at: https://github.com/isbrycee/GEM