Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models

TL;DR

This paper introduces a governance-aware hybrid fine-tuning method for multilingual large language models, improving adaptation accuracy and calibration while maintaining efficiency and stability across low-resource languages.

Contribution

It proposes a novel hybrid fine-tuning framework combining gradient-aligned low-rank updates with structured orthogonal transformations, enhanced by lightweight data governance steps.

Findings

Consistent performance gains over PEFT baselines on XNLI and FLORES datasets.

Improved calibration and cross-language parity in multilingual models.

Enhanced resilience to orthographic variants and efficient training footprint.

Abstract

We present a governance-aware hybrid fine-tuning framework for multilingual, low-resource adaptation of large language models. The core algorithm combines gradient-aligned low-rank updates with structured orthogonal transformations through layer-wise mixing and introduces unitary constraints in selected sub-layers to stabilize deep optimization. In tandem with lightweight, label-free data governance steps, including language identification, near-duplicate removal, and quality filtering, the framework targets accuracy, calibration, and cross-language parity under tight compute budgets. Across XNLI and FLORES, the hybrid approach delivers consistent gains over strong PEFT baselines while maintaining directional balance and improving probability calibration, as shown in Tables II and III. It is more resilient to lightweight orthographic variants, as shown in Table IV, and benefits additively from simple governance steps, as shown in Table V. Training footprint measurements indicate modest overhead and a favorable cost-quality frontier, as shown in Table VI and Figure 2. Together, these results show that hybrid and unitary PEFT provide a stable and accessible path to resource-efficient multilingual adaptation when paired with practical data governance.