LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications

TL;DR

LiveSecBench is a continuously updated safety benchmark for Chinese language models, combining automated and human verification to evaluate safety across multiple dimensions and provide an up-to-date leaderboard.

Contribution

It introduces a dynamic safety benchmark with a novel dataset construction pipeline and an evaluation system for Chinese LLM safety assessment.

Findings

Evaluated 57 Chinese LLMs using ELO rating system.

Provides an up-to-date safety leaderboard.

Includes assessments across five safety dimensions.

Abstract

We introduce LiveSecBench, a continuously updated safety benchmark specifically for Chinese-language LLM application scenarios. LiveSecBench constructs a high-quality and unique dataset through a pipeline that combines automated generation with human verification. By periodically releasing new versions to expand the dataset and update evaluation metrics, LiveSecBench provides a robust and up-to-date standard for AI safety. In this report, we introduce our second release v251215, which evaluates across five dimensions (Public Safety, Fairness & Bias, Privacy, Truthfulness, and Mental Health Safety.) We evaluate 57 representative LLMs using an ELO rating system, offering a leaderboard of the current state of Chinese LLM safety. The result is available at https://livesecbench.intokentech.cn/.