# MFMINet: Multimodal fusion and cross-layer interaction network for semantic segmentation of high-resolution remote sensing images

## TL;DR

MFMINet is a new AI model that improves the accuracy of analyzing satellite images by combining different types of data and using advanced techniques to better understand complex scenes.

## Contribution

The novel MFMINet architecture introduces multimodal fusion and cross-layer interaction modules to enhance semantic segmentation of remote sensing images.

## Key findings

- MFMINet achieves mIoU scores of 89.96% and 88.24% on the ISPRS Vaihingen and Potsdam datasets.
- The MCFM module effectively integrates high-level and low-level multimodal features.
- SAM and DSAM modules improve global and multiscale context modeling.

## Abstract

In recent years, semantic segmentation of remote sensing images using deep convolutional neural networks (CNNs) has seen rapid development in fields like urban planning and land cover analysis. However, reliance on a single imaging modality is often hampered by spectral ambiguity, the absence of elevation cues, and geometric confusion, limiting the discrimination of spectrally similar yet distinct categories like roads versus roofs. While multisource data fusion has emerged as a promising solution, effectively leveraging complementary information from multimodal features remains challenging. To address these challenges, we propose a multimodal fusion and multilayer interaction network (MFMINet), a two-way encoder-decoder network. Our model employs a multimodal cross-layer fusion module (MCFM) to integrate high-level semantic information with low-level spatial details, exploring the complementarities between different information modalities. Additionally, we introduce a self-attention module (SAM) to capture long-range spatial dependencies and refine fused features. Additionally, we develop a feature enhancement module (FEM) that intelligently selects between Transformer blocks for narrow channels and CNN blocks for wide channels, followed by point-wise convolution for optimal feature integration. Furthermore, we propose a dual spatial awareness module (DSAM) to mitigate downsampling effects and process global multiscale contextual information. Extensive experiments on ISPRS Vaihingen and Potsdam datasets demonstrate superior performance, with mIoU reaching 89.96% and 88.24%, respectively, validating the effectiveness of our method.

•MFMINet enables effective fusion of IRRG and nDSM, boosting segmentation accuracy•MCFM module strengthens fusion of IRRG and nDSM information•SAM and DSAM significantly improve global and multiscale context modeling•FEM module intelligently selects transformer or CNN for efficient feature aggregation

MFMINet enables effective fusion of IRRG and nDSM, boosting segmentation accuracy

MCFM module strengthens fusion of IRRG and nDSM information

SAM and DSAM significantly improve global and multiscale context modeling

FEM module intelligently selects transformer or CNN for efficient feature aggregation

Earth sciences; Remote sensing; Machine learning