Seg-LSTM: Performance of xLSTM for Semantic Segmentation of Remotely Sensed Images

TL;DR

This paper evaluates the effectiveness of the xLSTM model in semantic segmentation of remotely sensed images, finding it generally underperforms compared to Transformer-based models.

Contribution

First to assess Vision-LSTM's performance in remote sensing image segmentation using a novel encoder-decoder architecture called Seg-LSTM.

Findings

Vision-LSTM underperforms compared to Vision-Transformers.

Seg-LSTM provides a baseline for future improvements.

Study introduces a new evaluation framework for Vision-LSTM in segmentation.

Abstract

Recent advancements in autoregressive networks with linear complexity have driven significant research progress, demonstrating exceptional performance in large language models. A representative model is the Extended Long Short-Term Memory (xLSTM), which incorporates gating mechanisms and memory structures, performing comparably to Transformer architectures in long-sequence language tasks. Autoregressive networks such as xLSTM can utilize image serialization to extend their application to visual tasks such as classification and segmentation. Although existing studies have demonstrated Vision-LSTM's impressive results in image classification, its performance in image semantic segmentation remains unverified. Our study represents the first attempt to evaluate the effectiveness of Vision-LSTM in the semantic segmentation of remotely sensed images. This evaluation is based on a specifically designed encoder-decoder architecture named Seg-LSTM, and comparisons with state-of-the-art segmentation networks. Our study found that Vision-LSTM's performance in semantic segmentation was limited and generally inferior to Vision-Transformers-based and Vision-Mamba-based models in most comparative tests. Future research directions for enhancing Vision-LSTM are recommended. The source code is available from https://github.com/zhuqinfeng1999/Seg-LSTM.