Dynamic Guided and Domain Applicable Safeguards for Enhanced Security in Large Language Models

TL;DR

This paper presents G4D, a multi-agent framework that improves LLM safety by providing domain-aware, unbiased safety responses, effectively defending against jailbreak attacks while maintaining general utility.

Contribution

Introduction of G4D, a multi-agent safety framework that leverages external information for domain-specific and unbiased LLM safety responses, addressing current defense limitations.

Findings

G4D enhances robustness against jailbreak attacks in general and domain-specific scenarios.

G4D maintains LLM utility and responsiveness while improving safety.

Extensive experiments validate G4D's effectiveness across datasets.

Abstract

With the extensive deployment of Large Language Models (LLMs), ensuring their safety has become increasingly critical. However, existing defense methods often struggle with two key issues: (i) inadequate defense capabilities, particularly in domain-specific scenarios like chemistry, where a lack of specialized knowledge can lead to the generation of harmful responses to malicious queries. (ii) over-defensiveness, which compromises the general utility and responsiveness of LLMs. To mitigate these issues, we introduce a multi-agents-based defense framework, Guide for Defense (G4D), which leverages accurate external information to provide an unbiased summary of user intentions and analytically grounded safety response guidance. Extensive experiments on popular jailbreak attacks and benign datasets show that our G4D can enhance LLM's robustness against jailbreak attacks on general and domain-specific scenarios without compromising the model's general functionality.