Consensus-Aligned Neuron Efficient Fine-Tuning Large Language Models for Multi-Domain Machine Translation

TL;DR

This paper introduces a neuron-efficient fine-tuning method for large language models to improve multi-domain machine translation, effectively capturing domain-specific nuances and outperforming existing methods.

Contribution

It proposes a novel consensus-aligned neuron selection and fine-tuning framework that enhances domain adaptation in LLMs for machine translation.

Findings

Outperforms strong PEFT baselines on multiple translation domains

Achieves state-of-the-art results on German-English and Chinese-English translation

Effectively mitigates parameter interference and domain overfitting

Abstract

Multi-domain machine translation (MDMT) aims to build a unified model capable of translating content across diverse domains. Despite the impressive machine translation capabilities demonstrated by large language models (LLMs), domain adaptation still remains a challenge for LLMs. Existing MDMT methods such as in-context learning and parameter-efficient fine-tuning often suffer from domain shift, parameter interference and limited generalization. In this work, we propose a neuron-efficient fine-tuning framework for MDMT that identifies and updates consensus-aligned neurons within LLMs. These neurons are selected by maximizing the mutual information between neuron behavior and domain features, enabling LLMs to capture both generalizable translation patterns and domain-specific nuances. Our method then fine-tunes LLMs guided by these neurons, effectively mitigating parameter interference and domain-specific overfitting. Comprehensive experiments on three LLMs across ten German-English and Chinese-English translation domains evidence that our method consistently outperforms strong PEFT baselines on both seen and unseen domains, achieving state-of-the-art performance.