Are Large Pre-trained Vision Language Models Effective Construction Safety Inspectors?

TL;DR

This paper introduces ConstructionSite 10k, a comprehensive dataset for evaluating vision language models in construction safety inspection, highlighting their zero-shot and few-shot capabilities and the need for further training.

Contribution

The paper presents a new large-scale dataset with annotations for multiple safety inspection tasks, enabling better evaluation and development of VLMs in construction safety.

Findings

VLMs show strong zero-shot and few-shot generalization abilities.

Additional training improves VLM performance on construction safety tasks.

The dataset facilitates training and benchmarking of VLMs for construction safety inspection.

Abstract

Construction safety inspections typically involve a human inspector identifying safety concerns on-site. With the rise of powerful Vision Language Models (VLMs), researchers are exploring their use for tasks such as detecting safety rule violations from on-site images. However, there is a lack of open datasets to comprehensively evaluate and further fine-tune VLMs in construction safety inspection. Current applications of VLMs use small, supervised datasets, limiting their applicability in tasks they are not directly trained for. In this paper, we propose the ConstructionSite 10k, featuring 10,000 construction site images with annotations for three inter-connected tasks, including image captioning, safety rule violation visual question answering (VQA), and construction element visual grounding. Our subsequent evaluation of current state-of-the-art large pre-trained VLMs shows notable generalization abilities in zero-shot and few-shot settings, while additional training is needed to make them applicable to actual construction sites. This dataset allows researchers to train and evaluate their own VLMs with new architectures and techniques, providing a valuable benchmark for construction safety inspection.