# MSI-FusionNet: a multi-modal spectral-image fusion network for sorghum variety identification

## TL;DR

MSI-FusionNet combines spectral and image data to accurately identify 12 sorghum varieties, improving accuracy and efficiency for the Baijiu brewing industry.

## Contribution

MSI-FusionNet introduces a novel multi-modal fusion network for sorghum variety identification using spectral and image data.

## Key findings

- MSI-FusionNet achieves 93.61% accuracy in classifying 12 sorghum varieties.
- The model improves accuracy by 10.7% over spectral data and 29.91% over image data alone.
- Using ShuffleNetV2 reduces model complexity while maintaining high performance.

## Abstract

Compositional differences among sorghum varieties influence the brewing process, flavor characteristics, and overall quality of Baijiu. This study proposes a Multi-Modal Spectral-Image Fusion Network (MSI-FusionNet) data fusion model for rapid and accurate identification of sorghum varieties. This model integrates one-dimensional spectral data obtained through hyperspectral imaging with two-dimensional image data captured using industrial microscopes. The model identifies 12 sorghum varieties with an accuracy of 93.33 %. Compared with using spectral or image data alone, MSI-FusionNet improves accuracy by 11.11 % and 29.63 %, respectively. To balance performance and efficiency, various classic 2D convolutional neural network (2DCNN) architectures were evaluated. The MSI-FusionNet model with ShuffleNetV2 as the 2DCNN structure demonstrated superior efficiency, significantly reducing model complexity and computational cost while maintaining high accuracy. MSI-FusionNet offers an efficient and accurate solution for identifying sorghum varieties for liquor enterprises, supporting the stability of Baijiu flavor and quality, and providing valuable technical support for the brewing industry.

Unlabelled Image

•Developed MSI-FusionNet, a multi-modal fusion network integrating spectral and microscopic image data.•Achieved 93.61 % accuracy for classifying 12 sorghum varieties using multi-modal fusion.•Improved classification accuracy by 10.7 % over spectral data and 29.91 % over image data alone.•Incorporated ShuffleNetV2 to reduce model complexity while maintaining high performance.

Developed MSI-FusionNet, a multi-modal fusion network integrating spectral and microscopic image data.

Achieved 93.61 % accuracy for classifying 12 sorghum varieties using multi-modal fusion.

Improved classification accuracy by 10.7 % over spectral data and 29.91 % over image data alone.

Incorporated ShuffleNetV2 to reduce model complexity while maintaining high performance.