Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning

TL;DR

This paper systematically studies multi-disciplinary fine-tuning of large language models, revealing empirical laws that guide effective training across scientific domains for improved generalization.

Contribution

It introduces the first comprehensive analysis of multi-disciplinary LLM fine-tuning and formulates four empirical laws to optimize cross-domain learning.

Findings

Multi-disciplinary learning shows higher variability than single-discipline.

Four empirical laws guide effective multi-disciplinary fine-tuning.

Asymmetric LoRA-MoE achieves robust gains with minimal parameters.

Abstract

While large language models (LLMs) have achieved strong performance through fine-tuning within individual scientific domains, their learning dynamics in multi-disciplinary contexts remains poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. In this work, we present the first systematic study of multi-disciplinary LLM fine-tuning, constructing a five-discipline corpus and analyzing learning patterns of full fine-tuning, LoRA, LoRA-MoE, and LoRA compositions. Particularly, our study shows that multi-disciplinary learning is substantially more variable than single-discipline training and distills four consistent empirical laws: (1) Balance-then-Diversity: low-resource disciplines degrade performance unless mitigated via diversity-aware upsampling; (2) Merge-then-Align: restoring instruction-following ability is critical for cross-discipline synergy; (3) Optimize-then-Scale: parameter scaling offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE yields robust gains with minimal trainable parameters via shared low-rank projection. Together, these laws form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.