Paying Alignment Tax with Contrastive Learning

TL;DR

This paper introduces a contrastive learning framework that effectively reduces bias and toxicity in language models while maintaining factual accuracy and knowledge, overcoming trade-offs faced by existing methods.

Contribution

The paper presents a novel contrastive learning approach with dynamic loss scaling that improves bias mitigation and faithfulness preservation simultaneously.

Findings

Significant reduction in toxicity across multiple benchmarks.

Enhanced faithfulness and knowledge retention in models.

First method to improve bias and accuracy concurrently.

Abstract

Current debiasing approaches often result a degradation in model capabilities such as factual accuracy and knowledge retention. Through systematic evaluation across multiple benchmarks, we demonstrate that existing debiasing methods face fundamental trade-offs, particularly in smaller models, leading to reduced truthfulness, knowledge loss, or unintelligible outputs. To address these limitations, we propose a contrastive learning framework that learns through carefully constructed positive and negative examples. Our approach introduces contrast computation and dynamic loss scaling to balance bias mitigation with faithfulness preservation. Experimental results across multiple model scales demonstrate that our method achieves substantial improvements in both toxicity reduction and faithfulness preservation. Most importantly, we show that our framework is the first to consistently improve both metrics simultaneously, avoiding the capability degradation characteristic of existing approaches. These results suggest that explicit modeling of both positive and negative examples through contrastive learning could be a promising direction for reducing the alignment tax in language model debiasing.