LLM-based HSE Compliance Assessment: Benchmark, Performance, and Advancements

TL;DR

This paper introduces HSE-Bench, a benchmark dataset for evaluating LLMs in HSE compliance assessment, revealing current models' reliance on semantic matching and proposing a new prompting method, RoE, to improve reasoning accuracy.

Contribution

The paper presents the first benchmark dataset for LLM-based HSE compliance assessment and introduces RoE, a prompting technique that enhances reasoning capabilities of LLMs in this domain.

Findings

LLMs perform well but mainly rely on semantic matching.

Current LLM reasoning traces lack systematic legal reasoning.

RoE improves LLM decision accuracy by simulating expert reasoning.

Abstract

Health, Safety, and Environment (HSE) compliance assessment demands dynamic real-time decision-making under complicated regulations and complex human-machine-environment interactions. While large language models (LLMs) hold significant potential for decision intelligence and contextual dialogue, their capacity for domain-specific knowledge in HSE and structured legal reasoning remains underexplored. We introduce HSE-Bench, the first benchmark dataset designed to evaluate the HSE compliance assessment capabilities of LLM. HSE-Bench comprises over 1,000 manually curated questions drawn from regulations, court cases, safety exams, and fieldwork videos, and integrates a reasoning flow based on Issue spotting, rule Recall, rule Application, and rule Conclusion (IRAC) to assess the holistic reasoning pipeline. We conduct extensive evaluations on different prompting strategies and more than 10 LLMs, including foundation models, reasoning models and multimodal vision models. The results show that, although current LLMs achieve good performance, their capabilities largely rely on semantic matching rather than principled reasoning grounded in the underlying HSE compliance context. Moreover, their native reasoning trace lacks the systematic legal reasoning required for rigorous HSE compliance assessment. To alleviate these, we propose a new prompting technique, Reasoning of Expert (RoE), which guides LLMs to simulate the reasoning process of different experts for compliance assessment and reach a more accurate unified decision. We hope our study highlights reasoning gaps in LLMs for HSE compliance and inspires further research on related tasks.