Phare: A Safety Probe for Large Language Models

TL;DR

Phare is a comprehensive multilingual diagnostic framework designed to evaluate large language models across safety dimensions like hallucinations, biases, and harmful content, revealing systematic vulnerabilities and guiding improvements.

Contribution

It introduces a novel safety probing framework that identifies specific failure modes in LLMs, moving beyond performance metrics to enhance safety and trustworthiness.

Findings

Systematic vulnerabilities across safety dimensions

Patterns of biases and harmful content generation

Insights for improving LLM robustness and alignment

Abstract

Ensuring the safety of large language models (LLMs) is critical for responsible deployment, yet existing evaluations often prioritize performance over identifying failure modes. We introduce Phare, a multilingual diagnostic framework to probe and evaluate LLM behavior across three critical dimensions: hallucination and reliability, social biases, and harmful content generation. Our evaluation of 17 state-of-the-art LLMs reveals patterns of systematic vulnerabilities across all safety dimensions, including sycophancy, prompt sensitivity, and stereotype reproduction. By highlighting these specific failure modes rather than simply ranking models, Phare provides researchers and practitioners with actionable insights to build more robust, aligned, and trustworthy language systems.