Multimodal and Hyperspectral Dataset for Segmentation of Bulky Waste using VIS, IR, NIR, and Terahertz Imaging

TL;DR

This paper introduces a comprehensive dataset combining multiple imaging techniques to improve the classification and segmentation of bulky waste, especially distinguishing wood from non-wood materials.

Contribution

The novel contribution is a publicly available multimodal dataset with VIS, IR, NIR, and THz imaging for waste segmentation, including detailed annotations and benchmark tasks.

Findings

The dataset includes 56 registered scenes with 22,659 annotated patches for binary and subclass segmentation tasks.

Baseline performance using CNNs and fusion architectures is reported to establish reference metrics for future work.

The dataset includes challenging scenarios like occlusions and embedded metals to encourage robust multimodal approaches.

Abstract

This study presents an annotated multi-sensor, multimodal, and hyperspectral dataset designed to support deep learning-based classification and segmentation of bulky waste. The dataset comprises four distinct sensor modalities: high-resolution visible RGB images (VIS), hyperspectral near-infrared (NIR), temporally resolved thermal infrared (IR), and terahertz (THz) imaging with depth information, providing complementary multimodal information. An image registration process aligns all modalities to a common reference frame, enabling near pixel-precise fusion across sensors. WoodVIT contains 56 registered multi-sensor scenes, partitioned into 22,659 annotated patches with two main classes (wood and non-wood) and 16 subclass labels. It includes pixel-masks and patch-wise annotations to facilitate both segmentation and classification tasks. The primary benchmark task is binary discrimination of wood versus non-wood. The dataset also includes challenging scenarios involving occlusion and concealed contaminants (e.g., embedded metals) to motivate robust multimodal fusion approaches. We provide predefined train/validation/test splits and report baseline results using convolutional neural networks and fusion architectures to establish reference performance. WoodVIT is publicly available to support research on multi-sensor learning for waste sorting.

Figures