Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains

TL;DR

This paper introduces an adversarial alignment framework to improve value consistency in large language models for sensitive domains, using adversarial training and a new bilingual evaluation dataset.

Contribution

It proposes a novel adversarial training method and trains a Value-Consistent LLM for sensitive domains, with bilingual evaluation, demonstrating improved performance over existing models.

Findings

VC-LLM outperforms mainstream models in Chinese and English tests

Adversarial training enhances value consistency in LLMs

Constructed bilingual evaluation dataset for sensitive domain assessment

Abstract

With the wide application of large language models (LLMs), the problems of bias and value inconsistency in sensitive domains have gradually emerged, especially in terms of race, society and politics. In this paper, we propose an adversarial alignment framework, which enhances the value consistency of the model in sensitive domains through continued pre-training, instruction fine-tuning and adversarial training. In adversarial training, we use the Attacker to generate controversial queries, the Actor to generate responses with value consistency, and the Critic to filter and ensure response quality. Furthermore, we train a Value-Consistent Large Language Model, VC-LLM, for sensitive domains, and construct a bilingual evaluation dataset in Chinese and English. The experimental results show that VC-LLM performs better than the existing mainstream models in both Chinese and English tests, verifying the effectiveness of the method. Warning: This paper contains examples of LLMs that are offensive or harmful in nature.