Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications

TL;DR

This paper presents a practical framework for evaluating safety risks at the application level of LLM systems, emphasizing real-world deployment and operational safety considerations.

Contribution

It introduces a novel, actionable framework for assessing safety risks in LLM applications, bridging theoretical safety concepts with practical deployment challenges.

Findings

Framework validated through real-world deployment

Guidelines for developing safety risk taxonomies

Enhanced safety evaluation practices for LLM applications

Abstract

Most safety testing efforts for large language models (LLMs) today focus on evaluating foundation models. However, there is a growing need to evaluate safety at the application level, as components such as system prompts, retrieval pipelines, and guardrails introduce additional factors that significantly influence the overall safety of LLM applications. In this paper, we introduce a practical framework for evaluating application-level safety in LLM systems, validated through real-world deployment across multiple use cases within our organization. The framework consists of two parts: (1) principles for developing customized safety risk taxonomies, and (2) practices for evaluating safety risks in LLM applications. We illustrate how the proposed framework was applied in our internal pilot, providing a reference point for organizations seeking to scale their safety testing efforts. This work aims to bridge the gap between theoretical concepts in AI safety and the operational realities of safeguarding LLM applications in practice, offering actionable guidance for safe and scalable deployment.