Safe-Embed: Unveiling the Safety-Critical Knowledge of Sentence Encoders

TL;DR

This paper explores how sentence encoders can identify and classify unsafe prompts to improve the safety of large language models, introducing new datasets and metrics for evaluation.

Contribution

It introduces new datasets and the Categorical Purity metric to evaluate sentence encoders' ability to detect and classify unsafe prompts, highlighting their effectiveness and limitations.

Findings

Sentence encoders can distinguish safe from unsafe prompts.

Existing encoders have limitations in classifying all unsafe prompt types.

New datasets and metrics facilitate better evaluation of safety detection capabilities.

Abstract

Despite the impressive capabilities of Large Language Models (LLMs) in various tasks, their vulnerability to unsafe prompts remains a critical issue. These prompts can lead LLMs to generate responses on illegal or sensitive topics, posing a significant threat to their safe and ethical use. Existing approaches attempt to address this issue using classification models, but they have several drawbacks. With the increasing complexity of unsafe prompts, similarity search-based techniques that identify specific features of unsafe prompts provide a more robust and effective solution to this evolving problem. This paper investigates the potential of sentence encoders to distinguish safe from unsafe prompts, and the ability to classify various unsafe prompts according to a safety taxonomy. We introduce new pairwise datasets and the Categorical Purity (CP) metric to measure this capability. Our findings reveal both the effectiveness and limitations of existing sentence encoders, proposing directions to improve sentence encoders to operate as more robust safety detectors. Our code is available at https://github.com/JwdanielJung/Safe-Embed.