Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation

TL;DR

This paper introduces Orthogonal Gradient Selection (OGS), a novel data selection method that improves domain adaptation of large language models by ensuring gradient orthogonality, leading to better domain performance and efficiency without costly gradient projections.

Contribution

OGS shifts gradient safety from the optimizer to data selection using reinforcement learning and a lightweight Navigator, enabling efficient domain adaptation without incurring high computational costs.

Findings

OGS significantly improves domain-specific performance across multiple fields.

OGS maintains or enhances general reasoning capabilities like GSM8K.

OGS reduces training time compared to gradient projection methods.

Abstract

Fine-tuning large language models (LLMs) for specialized domains often necessitates a trade-off between acquiring domain expertise and retaining general reasoning capabilities, a phenomenon known as catastrophic forgetting. Existing remedies face a dichotomy: gradient surgery methods offer geometric safety but incur prohibitive computational costs via online projections, while efficient data selection approaches reduce overhead but remain blind to conflict-inducing gradient directions. In this paper, we propose Orthogonal Gradient Selection (OGS), a data-centric method that harmonizes domain performance, general capability retention, and training efficiency. OGS shifts the geometric insights of gradient projection from the optimizer to the data selection stage by treating data selection as a constrained decision-making process. By leveraging a lightweight Navigator model and reinforcement learning techniques, OGS dynamically identifies training samples whose gradients are orthogonal to a general-knowledge anchor. This approach ensures naturally safe updates for target models without modifying the optimizer or incurring runtime projection costs. Experiments across medical, legal, and financial domains demonstrate that OGS achieves excellent results, significantly improving domain performance and training efficiency while maintaining or even enhancing performance on general tasks such as GSM8K.