Bridging the Experimental Last Mile: Digitizing Laboratory Know-How for Safe AI-Assisted Support

TL;DR

This paper presents a human-in-the-loop AI assistant that uses multimodal data and retrieval-augmented generation to capture and communicate site-specific laboratory knowledge, enhancing safety and reliability in experimental settings.

Contribution

It introduces a novel framework combining video, multimodal AI, and RAG to extract and deliver lab-specific know-how, addressing the gap between documentation and practice.

Findings

System accurately extracts procedural knowledge from videos.

The AI assistant provides safe, grounded responses aligned with manuals.

Expert evaluation confirms usefulness and safety of generated advice.

Abstract

While advances in materials informatics have accelerated the development of Self-Driving Laboratories (SDLs), human-led experiments remain standard in many educational and exploratory research laboratories. In specific lab settings, formal documentation alone is often insufficient for safe and reliable operation. We refer to the gap between formal documentation and reliable execution in such settings as the experimental last mile; this gap mainly involves site-specific operational know-how, including local rules, routine checks, procedural details, and safety-conscious actions that are can be verbalizable but are often under-documented in standard manuals. In this proof-of-concept study, we developed a human-in-the-loop AI assistant that combines first-person experimental video, multimodal AI, and retrieval-augmented generation (RAG). Using powder X-ray diffraction experiments and student-recorded video data as inputs, the system extracts site-specific laboratory knowledge from recorded procedures, including physical techniques and audible confirmation that conventional manuals could omit. It then provides grounded responses based on the resulting manual. To reduce the risk of unsupported outputs, the system employs a two-layer safety design: source restriction through RAG and strict system-prompt constraints. Instructor-based evaluation showed alignment with expected guidance for questions covered by the manual. For out-of-scope queries, the system appropriately refused to answer, indicating a reduced risk of hallucination. Expert evaluation further indicated that the generated advisory reports were useful and safe (utility: 3.25/4.00; safety: 4.00/4.00). These results suggest the feasibility of a framework for bridging the experimental last mile in which AI supports laboratory practice under explicit human supervision rather than replacing human judgment.