Automated Hazard Detection in Construction Sites Using Large Language and Vision-Language Models

TL;DR

This thesis develops a multimodal AI framework combining text and image analysis to improve hazard detection on construction sites, demonstrating the effectiveness of lightweight models in safety monitoring.

Contribution

It introduces a novel multimodal AI framework for construction safety hazard detection, utilizing both large language models and vision-language models, including lightweight open-source options.

Findings

Lightweight models like Molmo 7B and Qwen2 VL 2B perform competitively in hazard detection tasks.

The hybrid pipeline effectively extracts structured insights from safety reports.

Cost-effective models can be used for large-scale safety monitoring.

Abstract

This thesis explores a multimodal AI framework for enhancing construction safety through the combined analysis of textual and visual data. In safety-critical environments such as construction sites, accident data often exists in multiple formats, such as written reports, inspection records, and site imagery, making it challenging to synthesize hazards using traditional approaches. To address this, this thesis proposed a multimodal AI framework that combines text and image analysis to assist in identifying safety hazards on construction sites. Two case studies were consucted to evaluate the capabilities of large language models (LLMs) and vision-language models (VLMs) for automated hazard identification.The first case study introduces a hybrid pipeline that utilizes GPT 4o and GPT 4o mini to extract structured insights from a dataset of 28,000 OSHA accident reports (2000-2025). The second case study extends this investigation using Molmo 7B and Qwen2 VL 2B, lightweight, open-source VLMs. Using the public ConstructionSite10k dataset, the performance of the two models was evaluated on rule-level safety violation detection using natural language prompts. This experiment served as a cost-aware benchmark against proprietary models and allowed testing at scale with ground-truth labels. Despite their smaller size, Molmo 7B and Quen2 VL 2B showed competitive performance in certain prompt configurations, reinforcing the feasibility of low-resource multimodal systems for rule-aware safety monitoring.