Soft Alignment Objectives for Robust Adaptation of Language Generation

TL;DR

This paper proposes novel soft alignment training objectives for domain adaptation in language models, improving robustness and reducing catastrophic forgetting without significant computational overhead.

Contribution

Introduces semantic similarity-based training objectives that mitigate forgetting during domain adaptation while maintaining model quality and efficiency.

Findings

Mitigates catastrophic forgetting in domain adaptation.

Preserves language model quality during adaptation.

Adds negligible computational costs.

Abstract

Domain adaptation allows generative language models to address specific flaws caused by the domain shift of their application. However, the traditional adaptation by further training on in-domain data rapidly weakens the model's ability to generalize to other domains, making the open-ended deployments of the adapted models prone to errors. This work introduces novel training objectives built upon a semantic similarity of the predicted tokens to the reference. Our results show that (1) avoiding the common assumption of a single correct prediction by constructing the training target from tokens' semantic similarity can mitigate catastrophic forgetting during domain adaptation, while (2) preserving the quality of the adaptation, (3) with negligible additions to compute costs. In the broader context, the objectives grounded in a continuous token similarity pioneer the exploration of the middle ground between the efficient but na\"{\i}ve exact-match token-level objectives and expressive but computationally- and resource-intensive sequential objectives.