The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models

TL;DR

This paper introduces the Safety Gap Toolkit, an open-source framework to evaluate the increase in dangerous capabilities of large language models when safeguards are removed, highlighting risks as models scale up.

Contribution

The paper presents a novel toolkit for estimating safety gaps in open-source LLMs and provides empirical analysis across different models and safeguard removal techniques.

Findings

Safety gap widens with model scale

Dangerous capabilities increase after safeguard removal

Effective safeguards significantly reduce risks

Abstract

Open-weight large language models (LLMs) unlock huge benefits in innovation, personalization, privacy, and democratization. However, their core advantage - modifiability - opens the door to systemic risks: bad actors can trivially subvert current safeguards, turning beneficial models into tools for harm. This leads to a 'safety gap': the difference in dangerous capabilities between a model with intact safeguards and one that has been stripped of those safeguards. We open-source a toolkit to estimate the safety gap for state-of-the-art open-weight models. As a case study, we evaluate biochemical and cyber capabilities, refusal rates, and generation quality of models from two families (Llama-3 and Qwen-2.5) across a range of parameter scales (0.5B to 405B) using different safeguard removal techniques. Our experiments reveal that the safety gap widens as model scale increases and effective dangerous capabilities grow substantially when safeguards are removed. We hope that the Safety Gap Toolkit (https://github.com/AlignmentResearch/safety-gap) will serve as an evaluation framework for common open-source models and as a motivation for developing and testing tamper-resistant safeguards. We welcome contributions to the toolkit from the community.