MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance

TL;DR

MAL introduces a novel pretraining framework for xLSTM that leverages cluster-masked and multi-task strategies, significantly improving performance and robustness across various visual tasks.

Contribution

The paper presents a new pretraining approach combining cluster-masked masking and multi-task learning to enhance xLSTM's capabilities in visual computing.

Findings

MAL outperforms traditional supervised models in visual tasks.

The framework improves local feature capture and image scanning efficiency.

MAL sets a new benchmark in xLSTM-based visual performance.

Abstract

The Long Short-Term Memory (LSTM) networks have traditionally faced challenges in scaling and effectively capturing complex dependencies in visual tasks. The xLSTM architecture has emerged to address these limitations, incorporating exponential gating and a parallel matrix memory structure to enhance performance and scalability. Despite these advancements, the potential of xLSTM in visual computing has not been fully realized, particularly in leveraging autoregressive techniques for improved feature extraction. In this paper, we introduce MAL (Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance), a novel framework that enhances xLSTM's capabilities through innovative pretraining strategies. We propose a cluster-masked masking method that significantly improves local feature capture and optimizes image scanning efficiency. Additionally, our universal encoder-decoder pretraining approach integrates multiple tasks, including image autoregression, depth estimation, and image segmentation, thereby enhancing the model's adaptability and robustness across diverse visual tasks. Our experimental results demonstrate that MAL surpasses traditional supervised models and fully leverages the scaling potential of xLSTM, setting a new benchmark in visual task performance.