VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis

TL;DR

VSLLaVA is a specialized large multimodal model pipeline designed for industrial vibration signal analysis, leveraging expert knowledge, a novel dataset, and tailored training and evaluation methods to improve signal classification and fault detection.

Contribution

The paper introduces VSLLaVA, a comprehensive pipeline combining expert-guided instruction tuning, a novel dataset, and specialized evaluation for industrial vibration analysis, advancing domain-specific large multimodal models.

Findings

Significantly improves signal type identification accuracy.

Enhances fault-related signal analysis capabilities.

Demonstrates effective domain-specific model development.

Abstract

While Large Multimodal Models (LMMs) excel in general multimodal tasks, they lack the domain-specific knowledge for industrial vibration signal analysis. This paper introduces VSLLaVA, a comprehensive pipeline that utilizes expert knowledge-guided instruction tuning and evaluation to create an end-to-end LMM for signal analysis. To achieve this, we construct a novel Signal-Question-Answer (SQA) dataset using an expert rule-based signal generator. This dataset facilitates a two-stage learning procedure. The first step is efficient instruction fine-tuning with Low-Rank Adaptation (LoRA), which imparts specialized signal identification capabilities. Subsequently, we designed a tailored Group Relative Policy Optimization (GRPO) to refine the reasoning capabilities and enhance classification robustness. Then, a dual-mode evaluation framework is proposed, combining an LLM referee with expert rules for semantic assessment using quantitative metrics for numerical and textual accuracy, which reveals that VSLLaVA significantly improves performance in signal type identification and parameter analysis, and makes progress in the identification and parameter analysis of fault-related signals. This research demonstrates a viable approach for developing specialized foundational models for complex industrial applications and marks a transition from conventional task-specific systems to a cohesive, interactive foundational model.