Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

TL;DR

The paper introduces DS-MoE, a text-guided dynamic sparse expert framework that enhances visual defect recognition by aligning textual semantics with visual patterns, achieving superior accuracy and efficiency.

Contribution

It presents a novel LLM-driven sparse MoE architecture with dynamic routing and lightweight encoding for improved defect detection across diverse datasets.

Findings

Outperforms existing vision models on multiple defect datasets.

Achieves +13.9, +1.4, +2.0 pp mAP improvements over YOLOv8/YOLOX.

Enables real-time inference with high accuracy.

Abstract

High inter-class similarity, extreme scale variation, and limited computational budgets hinder reliable visual recognition across diverse real-world data. Existing vision-centric and cross-modal approaches often rely on rigid fusion mechanisms and heavy annotation pipelines, leading to sub-optimal generalization. We propose the Distilled Large Language Model (LLM)-Driven Sparse Mixture-of-Experts (DS-MoE) framework, which integrates text-guided dynamic routing and lightweight multi-scale comprehension. The DS-MoE framework dynamically aligns textual semantics with defect-specific visual patterns through a sparse MoE architecture, where task-relevant experts are adaptively activated based on semantic relevance, resolving inter-class ambiguity. A lightweight MobileSAM encoder enables real-time inference while preserving multi-scale defect details. Extensive experiments on PCB, aluminum foil, and mold defect datasets demonstrate that our framework achieves superior performance compared to existing pure vision models. \textbf{DS-MoE} surpasses YOLOv8/YOLOX with gains of +13.9, +1.4, and +2.0 pp mAP@ 0.5:0.95 on BBMP, aluminum, and PCB, respectively, while also improving precision and recall.