SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds from RGB Images for 2D Classification

TL;DR

SIM-Net introduces a novel multimodal architecture that infers 3D point clouds from RGB images to improve 2D classification, especially for challenging herbarium specimen images with occlusions and heterogeneous backgrounds.

Contribution

The paper presents a new pixel-to-point transformation method and a fusion architecture combining 2D and 3D features for enhanced image classification performance.

Findings

Outperforms ResNet101 with up to 9.9% accuracy gain

Achieves 12.3% higher F-score over baseline models

Surpasses transformer-based architectures in herbarium specimen classification

Abstract

We introduce the Shape-Image Multimodal Network (SIM-Net), a novel 2D image classification architecture that integrates 3D point cloud representations inferred directly from RGB images. Our key contribution lies in a pixel-to-point transformation that converts 2D object masks into 3D point clouds, enabling the fusion of texture-based and geometric features for enhanced classification performance. SIM-Net is particularly well-suited for the classification of digitized herbarium specimens (a task made challenging by heterogeneous backgrounds), non-plant elements, and occlusions that compromise conventional image-based models. To address these issues, SIM-Net employs a segmentation-based preprocessing step to extract object masks prior to 3D point cloud generation. The architecture comprises a CNN encoder for 2D image features and a PointNet-based encoder for geometric features, which are fused into a unified latent space. Experimental evaluations on herbarium datasets demonstrate that SIM-Net consistently outperforms ResNet101, achieving gains of up to 9.9% in accuracy and 12.3% in F-score. It also surpasses several transformer-based state-of-the-art architectures, highlighting the benefits of incorporating 3D structural reasoning into 2D image classification tasks.