Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structure

TL;DR

This paper explores using vision language models for autonomous safety monitoring in laboratories, introducing a structured data pipeline and a scene-graph-guided alignment method to improve hazard detection from visual data.

Contribution

It presents a novel data generation pipeline for aligning images, scene graphs, and text, and proposes a scene-graph-guided alignment technique to enhance VLMs' hazard detection capabilities.

Findings

VLMs perform well with textual scene graphs.

Performance drops significantly with visual-only inputs.

Scene-graph-guided alignment improves hazard detection from images.

Abstract

Laboratories are prone to severe injuries from minor unsafe actions, yet continuous safety monitoring -- beyond mandatory pre-lab safety training -- is limited by human availability. Vision language models (VLMs) offer promise for autonomous laboratory safety monitoring, but their effectiveness in realistic settings is unclear due to the lack of visual evaluation data, as most safety incidents are documented primarily as unstructured text. To address this gap, we first introduce a structured data generation pipeline that converts textual laboratory scenarios into aligned triples of (image, scene graph, ground truth), using large language models as scene graph architects and image generation models as renderers. Our experiments on the synthetic dataset of 1,207 samples across 362 unique scenarios and seven open- and closed-source models show that VLMs perform effectively given textual scene graph, but degrade substantially in visual-only settings indicating difficulty in extracting structured object relationships directly from pixels. To overcome this, we propose a post-training context-engineering approach, scene-graph-guided alignment, to bridge perceptual gaps in VLMs by translating visual inputs into structured scene graphs better aligned with VLM reasoning, improving hazard detection performance in visual only settings.