Glass Segmentation with RGB-Thermal Image Pairs

TL;DR

This paper introduces a novel neural network approach that combines RGB and thermal images using attention-based fusion to improve glass segmentation accuracy, supported by a new dataset and extensive evaluations.

Contribution

The paper presents a new multi-modal fusion network architecture and a dataset for glass segmentation using RGB-thermal image pairs, enhancing segmentation performance.

Findings

Effective fusion of RGB and thermal data improves glass segmentation.

The proposed method outperforms existing approaches in accuracy.

A new dataset with 5551 image pairs supports robust evaluation.

Abstract

This paper proposes a new glass segmentation method utilizing paired RGB and thermal images. Due to the large difference between the transmission property of visible light and that of the thermal energy through the glass where most glass is transparent to the visible light but opaque to thermal energy, glass regions of a scene are made more distinguishable with a pair of RGB and thermal images than solely with an RGB image. To exploit such a unique property, we propose a neural network architecture that effectively combines an RGB-thermal image pair with a new multi-modal fusion module based on attention, and integrate CNN and transformer to extract local features and non-local dependencies, respectively. As well, we have collected a new dataset containing 5551 RGB-thermal image pairs with ground-truth segmentation annotations. The qualitative and quantitative evaluations demonstrate the effectiveness of the proposed approach on fusing RGB and thermal data for glass segmentation. Our code and data are available at https://github.com/Dong-Huo/RGB-T-Glass-Segmentation.