Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis

TL;DR

This paper introduces a novel, cost-effective framework that uses vision-language instruction tuning with large pre-trained models to analyze electron microscopy images in semiconductor manufacturing, reducing the need for extensive human labeling.

Contribution

It presents a new teacher-student approach leveraging GPT-4 for instruction data generation, enabling small multimodal models to perform microscopy analysis with minimal labeled data.

Findings

Effective zero-shot VQA and classification on microscopy images

Reduced human labeling through knowledge transfer from large models

Cost-effective and customizable analysis framework

Abstract

We present a novel framework for analyzing and interpreting electron microscopy images in semiconductor manufacturing using vision-language instruction tuning. The framework employs a unique teacher-student approach, leveraging pre-trained multimodal large language models such as GPT-4 to generate instruction-following data for zero-shot visual question answering (VQA) and classification tasks, customizing smaller multimodal models (SMMs) for microscopy image analysis, resulting in an instruction-tuned language-and-vision assistant. Our framework merges knowledge engineering with machine learning to integrate domain-specific expertise from larger to smaller multimodal models within this specialized field, greatly reducing the need for extensive human labeling. Our study presents a secure, cost-effective, and customizable approach for analyzing microscopy images, addressing the challenges of adopting proprietary models in semiconductor manufacturing.