FuseUNet: A Multi-Scale Feature Fusion Method for U-like Networks

TL;DR

FuseUNet introduces a novel multi-scale feature fusion approach for U-shaped networks, addressing limitations of traditional skip connections by modeling them as an initial value problem solved via an adaptive differential equation, enhancing feature integration.

Contribution

It proposes a differential equation-based multi-scale feature fusion method for U-Net architectures, improving feature interaction and efficiency without altering core encoder-decoder structures.

Findings

Improved segmentation accuracy on multiple datasets.

Reduced network parameters while maintaining performance.

Enhanced multi-scale feature utilization.

Abstract

Medical image segmentation is a critical task in computer vision, with UNet serving as a milestone architecture. The typical component of UNet family is the skip connection, however, their skip connections face two significant limitations: (1) they lack effective interaction between features at different scales, and (2) they rely on simple concatenation or addition operations, which constrain efficient information integration. While recent improvements to UNet have focused on enhancing encoder and decoder capabilities, these limitations remain overlooked. To overcome these challenges, we propose a novel multi-scale feature fusion method that reimagines the UNet decoding process as solving an initial value problem (IVP), treating skip connections as discrete nodes. By leveraging principles from the linear multistep method, we propose an adaptive ordinary differential equation method to enable effective multi-scale feature fusion. Our approach is independent of the encoder and decoder architectures, making it adaptable to various U-Net-like networks. Experiments on ACDC, KiTS2023, MSD brain tumor, and ISIC2017/2018 skin lesion segmentation datasets demonstrate improved feature utilization, reduced network parameters, and maintained high performance. The code is available at https://github.com/nayutayuki/FuseUNet.