QuPAINT: Physics-Aware Instruction Tuning Approach to Quantum Material Discovery

TL;DR

This paper introduces QuPAINT, a physics-aware multimodal framework for quantum material discovery that combines synthetic data generation, instruction tuning, and a new benchmark to improve generalization and interpretability in optical microscopy analysis.

Contribution

It presents Synthia, a physics-based synthetic data generator; QMat-Instruct, a large-scale instruction dataset; and QuPAINT, a physics-aware instruction tuning method with a novel attention module, advancing quantum material analysis.

Findings

Enhanced generalization across labs and materials.

Reduced need for manual annotations.

Improved robustness of quantum material classification.

Abstract

Characterizing two-dimensional quantum materials from optical microscopy images is challenging due to the subtle layer-dependent contrast, limited labeled data, and significant variation across laboratories and imaging setups. Existing vision models struggle in this domain since they lack physical priors and cannot generalize to new materials or hardware conditions. This work presents a new physics-aware multimodal framework that addresses these limitations from both the data and model perspectives. We first present Synthia, a physics-based synthetic data generator that simulates realistic optical responses of quantum material flakes under thin-film interference. Synthia produces diverse and high-quality samples, helping reduce the dependence on expert manual annotation. We introduce QMat-Instruct, the first large-scale instruction dataset for quantum materials, comprising multimodal, physics-informed question-answer pairs designed to teach Multimodal Large Language Models (MLLMs) to understand the appearance and thickness of flakes. Then, we propose Physics-Aware Instruction Tuning (QuPAINT), a multimodal architecture that incorporates a Physics-Informed Attention module to fuse visual embeddings with optical priors, enabling more robust and discriminative flake representations. Finally, we establish QF-Bench, a comprehensive benchmark spanning multiple materials, substrates, and imaging settings, offering standardized protocols for fair and reproducible evaluation.