Preliminary Investigations of a Multi-Faceted Robust and Synergistic Approach in Semiconductor Electron Micrograph Analysis: Integrating Vision Transformers with Large Language and Multimodal Models

TL;DR

This paper proposes a novel multi-faceted approach combining vision transformers, large language models, and multimodal models to improve the accuracy and robustness of semiconductor electron micrograph analysis for nanomaterial classification.

Contribution

It introduces an integrated architecture that leverages zero-shot prompting, few-shot learning, and multimodal fusion to enhance micrograph classification in semiconductor research.

Findings

Outperforms traditional classification methods in accuracy.

Provides a robust and interpretable nanomaterial identification process.

Facilitates high-throughput screening in semiconductor manufacturing.

Abstract

Characterizing materials using electron micrographs is crucial in areas such as semiconductors and quantum materials. Traditional classification methods falter due to the intricatestructures of these micrographs. This study introduces an innovative architecture that leverages the generative capabilities of zero-shot prompting in Large Language Models (LLMs) such as GPT-4(language only), the predictive ability of few-shot (in-context) learning in Large Multimodal Models (LMMs) such as GPT-4(V)ision, and fuses knowledge across image based and linguistic insights for accurate nanomaterial category prediction. This comprehensive approach aims to provide a robust solution for the automated nanomaterial identification task in semiconductor manufacturing, blending performance, efficiency, and interpretability. Our method surpasses conventional approaches, offering precise nanomaterial identification and facilitating high-throughput screening.