Large model retrieval enhancement framework for construction site risk identification

TL;DR

This paper introduces a retrieval-augmented framework that enhances large language models for construction site hazard identification by integrating external knowledge and similar case retrieval, significantly improving accuracy without fine-tuning.

Contribution

It presents a novel retrieval-augmented approach combining external knowledge and case retrieval to improve LLM-based hazard identification in construction sites without fine-tuning.

Findings

Boosted GLM-4V accuracy to 50%

Achieved 35.49% improvement over baselines

Validated effectiveness of image retrieval strategy

Abstract

This study addresses construction site hazard identification by proposing a retrieval-augmented framework that enhances large language models (LLMs) without requiring fine-tuning. Current LLM-based approaches face limitations: image-text matching struggles with complex hazards, while instruction tuning lacks generalization and is resource-intensive. Our method dynamically integrates external knowledge and retrieved similar cases via prompt tuning, overcoming LLMs' limitations in domain knowledge and feature correlation. The framework comprises a case database, an image retrieval module, and an LLM-based reasoning module. Evaluated on real-site data, our approach boosted GLM-4V's accuracy to 50%, a 35.49% improvement over baselines, with consistent gains across hazard types. Ablation studies validated the effectiveness of our image retrieval strategy, showing the superiority of our LPIPS- and CLIP-based method. The proposed technique significantly improves identification accuracy and contextual understanding, demonstrating strong generalization and offering a practical path for intelligent safety risk detection in construction.