LISTA-Transformer Model Based on Sparse Coding and Attention Mechanism and Its Application in Fault Diagnosis

TL;DR

This paper introduces a novel LISTA-Transformer model that combines sparse coding and attention mechanisms, enhancing feature extraction for fault diagnosis and outperforming traditional and existing Transformer-based methods.

Contribution

The paper proposes a new LISTA-Transformer architecture integrating sparse coding with Transformers for improved local and global feature modeling in fault diagnosis.

Findings

Fault recognition rate reached 98.5% on CWRU dataset

Outperformed traditional methods by 3.3%

Demonstrated superiority over existing Transformer approaches

Abstract

Driven by the continuous development of models such as Multi-Layer Perceptron, Convolutional Neural Network (CNN), and Transformer, deep learning has made breakthrough progress in fields such as computer vision and natural language processing, and has been successfully applied in practical scenarios such as image classification and industrial fault diagnosis. However, existing models still have certain limitations in local feature modeling and global dependency capture. Specifically, CNN is limited by local receptive fields, while Transformer has shortcomings in effectively modeling local structures, and both face challenges of high model complexity and insufficient interpretability. In response to the above issues, we proposes the following innovative work: A sparse Transformer based on Learnable Iterative Shrinkage Threshold Algorithm (LISTA-Transformer) was designed, which deeply integrates LISTA sparse encoding with visual Transformer to construct a model architecture with adaptive local and global feature collaboration mechanism. This method utilizes continuous wavelet transform to convert vibration signals into time-frequency maps and inputs them into LISTA-Transformer for more effective feature extraction. On the CWRU dataset, the fault recognition rate of our method reached 98.5%, which is 3.3% higher than traditional methods and exhibits certain superiority over existing Transformer-based approaches.