SecureBreak -- A dataset towards safe and secure models

TL;DR

SecureBreak is a carefully annotated dataset aimed at improving the detection of unsafe outputs in large language models, enhancing security alignment and robustness against attacks like jailbreaking.

Contribution

The paper introduces SecureBreak, a new dataset for detecting harmful LLM outputs, supporting safety filtering and security enhancement efforts.

Findings

Fine-tuning on SecureBreak improves detection of unsafe content.

The dataset performs well across multiple risk categories.

SecureBreak aids in both post-generation filtering and model alignment.

Abstract

Large language models are becoming pervasive core components in many real-world applications. As a consequence, security alignment represents a critical requirement for their safe deployment. Although previous related works focused primarily on model architectures and alignment methodologies, these approaches alone cannot ensure the complete elimination of harmful generations. This concern is reinforced by the growing body of scientific literature showing that attacks, such as jailbreaking and prompt injection, can bypass existing security alignment mechanisms. As a consequence, additional security strategies are needed both to provide qualitative feedback on the robustness of the obtained security alignment at the training stage, and to create an ``ultimate'' defense layer to block unsafe outputs possibly produced by deployed models. To provide a contribution in this scenario, this paper introduces SecureBreak, a safety-oriented dataset designed to support the development of AI-driven solutions for detecting harmful LLM outputs caused by residual weaknesses in security alignment. The dataset is highly reliable due to careful manual annotation, where labels are assigned conservatively to ensure safety. It performs well in detecting unsafe content across multiple risk categories. Tests with pre-trained LLMs show improved results after fine-tuning on SecureBreak. Overall, the dataset is useful both for post-generation safety filtering and for guiding further model alignment and security improvements.