OmniCompliance-100K: A Multi-Domain, Rule-Grounded, Real-World Safety Compliance Dataset

TL;DR

This paper introduces OmniCompliance-100K, a large, rule-grounded safety dataset for LLMs, covering 74 regulations across multiple domains, and evaluates LLMs' safety capabilities using this dataset.

Contribution

It constructs a comprehensive, real-world compliance dataset with 12,985 rules and over 106,000 cases, addressing gaps in existing safety datasets and providing a benchmark for LLM safety evaluation.

Findings

Strong alignment between rules and cases confirmed

Benchmarking reveals variability in LLM safety across models

Insights suggest directions for future LLM safety improvements

Abstract

Ensuring the safety and compliance of large language models (LLMs) is of paramount importance. However, existing LLM safety datasets often rely on ad-hoc taxonomies for data generation and suffer from a significant shortage of rule-grounded, real-world cases that are essential for robustly protecting LLMs. In this work, we address this critical gap by constructing a comprehensive safety dataset from a compliance perspective. Using a powerful web-searching agent, we collect a rule-grounded, real-world case dataset OmniCompliance-100K, sourced from multi-domain authoritative references. The dataset spans 74 regulations and policies across a wide range of domains, including security and privacy regulations, content safety and user data privacy policies from leading AI companies and social media platforms, financial security requirements, medical device risk management standards, educational integrity guidelines, and protections of fundamental human rights. In total, our dataset contains 12,985 distinct rules and 106,009 associated real-world compliance cases. Our analysis confirms a strong alignment between the rules and their corresponding cases. We further conduct extensive benchmarking experiments to evaluate the safety and compliance capabilities of advanced LLMs across different model scales. Our experiments reveal several interesting findings that have great potential to offer valuable insights for future LLM safety research.