LJ-Bench: Ontology-Based Benchmark for U.S. Crime

TL;DR

LJ-Bench introduces an ontology-based benchmark grounded in U.S. legal frameworks to evaluate LLM robustness against a wide spectrum of illegal activities, addressing gaps in existing benchmarks.

Contribution

This work presents the first comprehensive, ontology-based benchmark for assessing LLM vulnerability to diverse crime categories grounded in legal standards.

Findings

LLMs are more vulnerable to societal harm attacks than individual harm attacks.

The benchmark covers 76 crime types organized taxonomically.

Experiments reveal specific weaknesses in LLMs across different crime categories.

Abstract

The potential of Large Language Models (LLMs) to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. Unfortunately, existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works. In this work, we introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code, which serves as an influential reference for criminal law and has been adopted by many U.S. states, and instantiated using Californian Law. This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities. Spanning 76 distinct crime types organized taxonomically, LJ-Bench enables systematic assessment of diverse attacks, revealing valuable insights into LLM vulnerabilities across various crime categories: LLMs exhibit heightened susceptibility to attacks targeting societal harm rather than those directly impacting individuals. Our benchmark aims to facilitate the development of more robust and trustworthy LLMs. The LJ-Bench benchmark and LJ-Ontology, along with experiments implementation for reproducibility are publicly available at https://github.com/AndreaTseng/LJ-Bench.