iSafetyBench: A video-language benchmark for safety in industrial environment

TL;DR

iSafetyBench is a new video-language benchmark designed to evaluate AI models' ability to recognize routine and hazardous activities in industrial environments, highlighting current models' limitations in safety-critical scenarios.

Contribution

The paper introduces iSafetyBench, a comprehensive dataset and benchmark for assessing vision-language models in industrial safety contexts, addressing a gap in existing video understanding benchmarks.

Findings

State-of-the-art models perform poorly on hazardous activity recognition.

Models struggle with multi-label and safety-critical scenarios.

Significant performance gaps highlight the need for safety-aware models.

Abstract

Recent advances in vision-language models (VLMs) have enabled impressive generalization across diverse video understanding tasks under zero-shot settings. However, their capabilities in high-stakes industrial domains-where recognizing both routine operations and safety-critical anomalies is essential-remain largely underexplored. To address this gap, we introduce iSafetyBench, a new video-language benchmark specifically designed to evaluate model performance in industrial environments across both normal and hazardous scenarios. iSafetyBench comprises 1,100 video clips sourced from real-world industrial settings, annotated with open-vocabulary, multi-label action tags spanning 98 routine and 67 hazardous action categories. Each clip is paired with multiple-choice questions for both single-label and multi-label evaluation, enabling fine-grained assessment of VLMs in both standard and safety-critical contexts. We evaluate eight state-of-the-art video-language models under zero-shot conditions. Despite their strong performance on existing video benchmarks, these models struggle with iSafetyBench-particularly in recognizing hazardous activities and in multi-label scenarios. Our results reveal significant performance gaps, underscoring the need for more robust, safety-aware multimodal models for industrial applications. iSafetyBench provides a first-of-its-kind testbed to drive progress in this direction. The dataset is available at: https://github.com/iSafetyBench/data.