DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training

TL;DR

This paper introduces DART, a training method to reduce harm drift in difference-aware large language models by distilling reasoning, auditing outputs, and repairing problematic cases, improving accuracy and safety.

Contribution

DART is a novel training framework that mitigates harm drift in LLMs by combining distillation, auditing, and targeted fine-tuning, enhancing difference-awareness and safety.

Findings

DART increased Llama-3-8B-Instruct accuracy from 39.0% to 68.8%.

Harm drift cases were reduced by 72.6%.

Difference-appropriate responses improved from 39.8% to 77.5%.

Abstract

Large language models (LLMs) tuned for safety often avoid acknowledging demographic differences, even when such acknowledgment is factually correct (e.g., ancestry-based disease incidence) or contextually justified (e.g., religious hiring preferences). This identity-blindness yields incorrect responses, unnecessary refusals, or generic "equal-treatment" defaults. We study this via difference-awareness classification: given a question involving demographic groups, the task is not to answer directly, but to classify whether a correct answer requires recognizing group differences (yes) or whether groups should be treated identically (no). Crucially, fine-tuning for accuracy triggers harm drift: model-generated explanations become increasingly harmful as decision accuracy improves, whether by elaborating harmful content, introducing problematic assumptions, or failing to flag harms the baseline identified. To mitigate this, we introduce DART (Distill--Audit--Repair Training), which distills label-conditioned reasoning from a teacher, audits outputs for harm drift cases relative to baseline, and repairs problematic cases via severity-weighted fine-tuning. On eight benchmarks, DART improves Llama-3-8B-Instruct accuracy from 39.0% to 68.8%, with largest gains on equal-treatment prompts (11.3% -> 72.6%), while reducing harm drift cases by 72.6%. It also transfers to 280 open-ended real-world queries across medical, legal, policy, and educational domains, improving difference-appropriate responses from 39.8% to 77.5% while reducing refusals from 34.3% to 3.0%. Our results demonstrate that accuracy and safety need not conflict when explicit detection and repair mechanisms are in place.